[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:14 dist-test-slave-8tdl krb5kdc[28091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:14 dist-test-slave-8tdl krb5kdc[28091](info): set up 2 sockets
Aug 11 03:04:14 dist-test-slave-8tdl krb5kdc[28091](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 03:04:15.353457 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:15 dist-test-slave-8tdl krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:04:15.567999 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34339
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:15.930660 28105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:15.931131 28105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:15.931310 28105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:15.939074 28105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:15.939210 28105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:15.939302 28105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:15.939407 28105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:15.939481 28105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:15.950309 28105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:15.951879 28105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:15.953320 28105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:15.967324 28115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:15.967619 28110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:15.967571 28111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:15.968600 28105 server_base.cc:1034] running on GCE node
I20260811 03:04:15.969646 28105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:15.971472 28105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:15.972776 28105 hybrid_clock.cc:648] HybridClock initialized: now 1786417455972720 us; error 141 us; skew 500 ppm
I20260811 03:04:15.973285 28105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:15 dist-test-slave-8tdl krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417455, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:15.977128 28105 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:15.978935 28105 webserver.cc:460] Webserver started at http://127.27.109.62:46539/ using document root <none> and password file <none>
I20260811 03:04:15.979722 28105 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:15.979876 28105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:15.980229 28105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:15.983640 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "a0033e953218497386e02832c241dc7a"
format_stamp: "Formatted at 2026-08-11 03:04:15 on dist-test-slave-8tdl"
server_key: "ab790583af9aaef2ecaf00d43f6302aa"
server_key_iv: "cc285ce46f11964dc24823af2bcb9284"
server_key_version: "encryptionkey@0"
I20260811 03:04:15.984490 28105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "a0033e953218497386e02832c241dc7a"
format_stamp: "Formatted at 2026-08-11 03:04:15 on dist-test-slave-8tdl"
server_key: "ab790583af9aaef2ecaf00d43f6302aa"
server_key_iv: "cc285ce46f11964dc24823af2bcb9284"
server_key_version: "encryptionkey@0"
I20260811 03:04:15.990440 28105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260811 03:04:15.995157 28122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:15.996716 28105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 03:04:15.996959 28105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "a0033e953218497386e02832c241dc7a"
format_stamp: "Formatted at 2026-08-11 03:04:15 on dist-test-slave-8tdl"
server_key: "ab790583af9aaef2ecaf00d43f6302aa"
server_key_iv: "cc285ce46f11964dc24823af2bcb9284"
server_key_version: "encryptionkey@0"
I20260811 03:04:15.997226 28105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:16.022258 28105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:16.027215 28105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:16.027550 28105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:16.048552 28105 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:34339
I20260811 03:04:16.048592 28178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:34339 every 8 connection(s)
I20260811 03:04:16.050137 28105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:16.054175 28179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:16.057193 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 28105
I20260811 03:04:16.057543 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:16.060215 28084 external_mini_cluster.cc:1428] Setting key 81532fa985b084d8c6852afe15492880
I20260811 03:04:16.068408 28179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a: Bootstrap starting.
I20260811 03:04:16.071614 28179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:16.072734 28179 log.cc:824] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:16.076104 28179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a: No bootstrap required, opened a new log
I20260811 03:04:16.085997 28179 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0033e953218497386e02832c241dc7a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34339 } }
I20260811 03:04:16.086513 28179 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:16.086627 28179 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0033e953218497386e02832c241dc7a, State: Initialized, Role: FOLLOWER
I20260811 03:04:16.087473 28179 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0033e953218497386e02832c241dc7a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34339 } }
I20260811 03:04:16.087805 28179 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:16.087983 28179 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:16.088177 28179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:16.090735 28179 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0033e953218497386e02832c241dc7a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34339 } }
I20260811 03:04:16.091322 28179 leader_election.cc:302] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0033e953218497386e02832c241dc7a; no voters:
I20260811 03:04:16.091998 28179 leader_election.cc:288] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:16.092191 28185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 1 FOLLOWER]: Leader election won for term 1
Aug 11 03:04:16 dist-test-slave-8tdl krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:16.093474 28185 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [term 1 LEADER]: Becoming Leader. State: Replica: a0033e953218497386e02832c241dc7a, State: Running, Role: LEADER
I20260811 03:04:16.094184 28185 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0033e953218497386e02832c241dc7a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34339 } }
I20260811 03:04:16.094422 28179 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:16.097405 28186 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0033e953218497386e02832c241dc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0033e953218497386e02832c241dc7a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34339 } } }
I20260811 03:04:16.097836 28186 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:16.099541 28187 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0033e953218497386e02832c241dc7a. Latest consensus state: current_term: 1 leader_uuid: "a0033e953218497386e02832c241dc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0033e953218497386e02832c241dc7a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34339 } } }
I20260811 03:04:16.099968 28187 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:16.105418 28191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:16.109954 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:16.065422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.27.109.62:34339)
0811 03:04:16.066492 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:16.066512 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:16.067962 (+ 1450us) server_negotiation.cc:407] Connection header received
0811 03:04:16.078135 (+ 10173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:16.078172 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:16.078774 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:16.079299 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:16.082305 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:16.084462 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:16.086340 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:16.087487 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:16.087718 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:16.095288 (+ 7570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:16.095303 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:16.095330 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:16.095393 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:16.101448 (+ 6055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:16.103188 (+ 1740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:16.103198 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:16.103206 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:16.103501 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:16.104334 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:16.104339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:16.104343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:16.105607 (+ 1264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:16.106050 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:16.107325 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0811 03:04:16.107919 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":486,"threads_started":1}
I20260811 03:04:16.113830 28191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:16.124989 28191 catalog_manager.cc:1347] Generated new cluster ID: a2b73e5598994c4e8edab042f0bdbc9d
I20260811 03:04:16.125149 28191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:16.141386 28191 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:16.142912 28191 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:16.151384 28191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0033e953218497386e02832c241dc7a: Generated new TSK 0
I20260811 03:04:16.152352 28191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:04:16.214402 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34339
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:16.612504 28207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:16.612828 28207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:16.612949 28207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:16.620982 28207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:16.621121 28207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:16.621245 28207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:16.631733 28207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:16.633162 28207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:16.634652 28207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:16.649786 28212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:16.649703 28213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:16.649811 28215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:16.650661 28207 server_base.cc:1034] running on GCE node
I20260811 03:04:16.651350 28207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:16.652410 28207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:16.653673 28207 hybrid_clock.cc:648] HybridClock initialized: now 1786417456653622 us; error 42 us; skew 500 ppm
I20260811 03:04:16.654052 28207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:16 dist-test-slave-8tdl krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417456, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:16.657691 28207 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:04:16.659499 28207 webserver.cc:460] Webserver started at http://127.27.109.1:46761/ using document root <none> and password file <none>
I20260811 03:04:16.660274 28207 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:16.660432 28207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:16.660780 28207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:16.664021 28207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "b7a253194a564b5ab296b784d6c29bc7"
format_stamp: "Formatted at 2026-08-11 03:04:16 on dist-test-slave-8tdl"
server_key: "9d3b18acd74fd4cb62414b2a2d79e275"
server_key_iv: "a7cbd874ea46c4c3a3ae65177e5e57b8"
server_key_version: "encryptionkey@0"
I20260811 03:04:16.664865 28207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "b7a253194a564b5ab296b784d6c29bc7"
format_stamp: "Formatted at 2026-08-11 03:04:16 on dist-test-slave-8tdl"
server_key: "9d3b18acd74fd4cb62414b2a2d79e275"
server_key_iv: "a7cbd874ea46c4c3a3ae65177e5e57b8"
server_key_version: "encryptionkey@0"
I20260811 03:04:16.671087 28207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 03:04:16.675632 28223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:16.677253 28207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:04:16.677495 28207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "b7a253194a564b5ab296b784d6c29bc7"
format_stamp: "Formatted at 2026-08-11 03:04:16 on dist-test-slave-8tdl"
server_key: "9d3b18acd74fd4cb62414b2a2d79e275"
server_key_iv: "a7cbd874ea46c4c3a3ae65177e5e57b8"
server_key_version: "encryptionkey@0"
I20260811 03:04:16.677773 28207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:16.708555 28207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:16.713236 28207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:16.713569 28207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:16.714792 28207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:16.716290 28207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:16.716369 28207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:16.716535 28207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:16.716612 28207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:16.753726 28207 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:40499
I20260811 03:04:16.753783 28351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:40499 every 8 connection(s)
I20260811 03:04:16.755228 28207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:16.761610 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 28207
I20260811 03:04:16.761878 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:16.762737 28084 external_mini_cluster.cc:1428] Setting key b7113286fd65fee1486b61000753c85f
Aug 11 03:04:16 dist-test-slave-8tdl krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417456, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:16.779923 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:16.758244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:43359 (local address 127.27.109.62:34339)
0811 03:04:16.758843 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:16.758848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:16.759703 (+ 855us) server_negotiation.cc:407] Connection header received
0811 03:04:16.762025 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:16.762033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:16.762194 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:16.762329 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:16.764433 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:16.765371 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:16.768241 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:16.768735 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:16.768880 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:16.773399 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:16.773405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:16.773409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:16.773447 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:16.776317 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:16.777180 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:16.777192 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:16.777195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:16.777323 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:16.777810 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:16.777814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:16.777817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:16.778110 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:16.778278 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:16.779254 (+ 976us) server_negotiation.cc:299] Negotiation successful
0811 03:04:16.779513 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":308,"threads_started":1}
I20260811 03:04:16.781011 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:16.758809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34339 (local address 127.27.109.1:43359)
0811 03:04:16.759533 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:16.759573 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:16.761617 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:16.762652 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:16.762665 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:16.763496 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:16.764256 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:16.764275 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:16.765551 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:16.765557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:16.768003 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:16.768026 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:16.769056 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:16.769065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:16.769446 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:16.770295 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:16.770338 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:16.773115 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:16.776495 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:16.776505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:16.776526 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:16.777028 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:16.777447 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:16.777452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:16.777456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:16.777673 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:16.778251 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:16.778259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:16.778726 (+ 467us) client_negotiation.cc:769] Sending connection context
0811 03:04:16.779134 (+ 408us) client_negotiation.cc:240] Negotiation successful
0811 03:04:16.779738 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":304,"threads_started":1}
I20260811 03:04:16.783233 28352 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34339
I20260811 03:04:16.783596 28352 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260811 03:04:16.784427 28352 heartbeater.cc:511] Master 127.27.109.62:34339 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260811 03:04:16.787297 28139 ts_manager.cc:194] Registered new tserver with Master: b7a253194a564b5ab296b784d6c29bc7 (127.27.109.1:40499)
I20260811 03:04:16.788887 28139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:43359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:04:16.814940 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34339
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:17.179150 28358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:17.179488 28358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:17.179602 28358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:17.188217 28358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:17.188369 28358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:17.188491 28358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:04:17.199101 28358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:17.200490 28358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:17.202011 28358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:17.217790 28366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:17.217968 28364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:17.217967 28363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:17.218742 28358 server_base.cc:1034] running on GCE node
I20260811 03:04:17.219496 28358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:17.220566 28358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:17.221829 28358 hybrid_clock.cc:648] HybridClock initialized: now 1786417457221775 us; error 44 us; skew 500 ppm
I20260811 03:04:17.222272 28358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:17 dist-test-slave-8tdl krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:17.231406 28358 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:04:17.233358 28358 webserver.cc:460] Webserver started at http://127.27.109.2:43709/ using document root <none> and password file <none>
I20260811 03:04:17.234175 28358 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:17.234355 28358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:17.234778 28358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:17.238119 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "98fd673744de444c8d3263c6d69ae444"
format_stamp: "Formatted at 2026-08-11 03:04:17 on dist-test-slave-8tdl"
server_key: "e729574be27b32a499449fd037ec617d"
server_key_iv: "c6beb50ac7e77735235e5a843a0b8e75"
server_key_version: "encryptionkey@0"
I20260811 03:04:17.239096 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "98fd673744de444c8d3263c6d69ae444"
format_stamp: "Formatted at 2026-08-11 03:04:17 on dist-test-slave-8tdl"
server_key: "e729574be27b32a499449fd037ec617d"
server_key_iv: "c6beb50ac7e77735235e5a843a0b8e75"
server_key_version: "encryptionkey@0"
I20260811 03:04:17.245301 28358 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:04:17.249364 28375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:17.250995 28358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 03:04:17.251214 28358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "98fd673744de444c8d3263c6d69ae444"
format_stamp: "Formatted at 2026-08-11 03:04:17 on dist-test-slave-8tdl"
server_key: "e729574be27b32a499449fd037ec617d"
server_key_iv: "c6beb50ac7e77735235e5a843a0b8e75"
server_key_version: "encryptionkey@0"
I20260811 03:04:17.251471 28358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:17.282908 28358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:17.287608 28358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:17.287972 28358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:17.289040 28358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:17.290575 28358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:17.290688 28358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:17.290874 28358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:17.290963 28358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:17.329227 28358 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:41269
I20260811 03:04:17.329277 28504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:41269 every 8 connection(s)
I20260811 03:04:17.330772 28358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:04:17.337335 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 28358
I20260811 03:04:17.337577 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:04:17.338044 28084 external_mini_cluster.cc:1428] Setting key cd037d61c851188eb36eb5fa1dc64b57
Aug 11 03:04:17 dist-test-slave-8tdl krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:17.354588 28507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:17.333741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:46715 (local address 127.27.109.62:34339)
0811 03:04:17.334328 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:17.334334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:17.335290 (+ 956us) server_negotiation.cc:407] Connection header received
0811 03:04:17.337273 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:17.337280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:17.337383 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:17.337511 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:17.339422 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.340362 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.342005 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.342473 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.342616 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:17.346525 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:17.346531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:17.346534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:17.346572 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:17.350418 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:17.351576 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:17.351585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:17.351591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:17.351786 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:17.352294 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:17.352302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:17.352307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:17.352765 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:17.352949 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:17.353752 (+ 803us) server_negotiation.cc:299] Negotiation successful
0811 03:04:17.354061 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":342,"threads_started":1}
I20260811 03:04:17.355512 28508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:17.334215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34339 (local address 127.27.109.2:46715)
0811 03:04:17.335112 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:17.335157 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:17.336958 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:17.337799 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:17.337808 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:17.338432 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:17.339237 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:17.339256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.340531 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.340535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:17.341833 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:17.341846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.342773 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.342782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:17.343037 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:17.343799 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:17.343824 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:17.346241 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:17.350767 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:17.350777 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:17.350795 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:17.351379 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:17.351915 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:17.351919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:17.351921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:17.352145 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:17.352915 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:17.352923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:17.353336 (+ 413us) client_negotiation.cc:769] Sending connection context
0811 03:04:17.353608 (+ 272us) client_negotiation.cc:240] Negotiation successful
0811 03:04:17.354072 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":349,"threads_started":1}
I20260811 03:04:17.357376 28505 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34339
I20260811 03:04:17.357758 28505 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:17.358868 28505 heartbeater.cc:511] Master 127.27.109.62:34339 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260811 03:04:17.360805 28139 ts_manager.cc:194] Registered new tserver with Master: 98fd673744de444c8d3263c6d69ae444 (127.27.109.2:41269)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260811 03:04:17.361925 28139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:46715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:04:17.387290 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34339
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:17.758774 28511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:17.759192 28511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:17.759373 28511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:17.768528 28511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:17.768662 28511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:17.768790 28511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:04:17.779419 28511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34339
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:17.780833 28511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:17.782313 28511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 03:04:17.793221 28352 heartbeater.cc:503] Master 127.27.109.62:34339 was elected leader, sending a full tablet report...
W20260811 03:04:17.796816 28518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:17.796921 28521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:17.797096 28519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:17.798496 28511 server_base.cc:1034] running on GCE node
I20260811 03:04:17.799276 28511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:17.800369 28511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:17.801630 28511 hybrid_clock.cc:648] HybridClock initialized: now 1786417457801509 us; error 86 us; skew 500 ppm
I20260811 03:04:17.801965 28511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:17 dist-test-slave-8tdl krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:17.805500 28511 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:04:17.807364 28511 webserver.cc:460] Webserver started at http://127.27.109.3:34871/ using document root <none> and password file <none>
I20260811 03:04:17.808200 28511 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:17.808362 28511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:17.808728 28511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:17.812211 28511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "1e132d9a63fd4d46914c407736bde419"
format_stamp: "Formatted at 2026-08-11 03:04:17 on dist-test-slave-8tdl"
server_key: "0c67f9ae90b00612830cba08a3b685c6"
server_key_iv: "e995262e800c92e576c5703337b20c43"
server_key_version: "encryptionkey@0"
I20260811 03:04:17.813069 28511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "1e132d9a63fd4d46914c407736bde419"
format_stamp: "Formatted at 2026-08-11 03:04:17 on dist-test-slave-8tdl"
server_key: "0c67f9ae90b00612830cba08a3b685c6"
server_key_iv: "e995262e800c92e576c5703337b20c43"
server_key_version: "encryptionkey@0"
I20260811 03:04:17.819650 28511 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260811 03:04:17.823671 28528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:17.825228 28511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:04:17.825454 28511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "1e132d9a63fd4d46914c407736bde419"
format_stamp: "Formatted at 2026-08-11 03:04:17 on dist-test-slave-8tdl"
server_key: "0c67f9ae90b00612830cba08a3b685c6"
server_key_iv: "e995262e800c92e576c5703337b20c43"
server_key_version: "encryptionkey@0"
I20260811 03:04:17.825793 28511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:17.845512 28511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:17.850638 28511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:17.851087 28511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:17.852216 28511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:17.853807 28511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:17.853921 28511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:17.854094 28511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:17.854171 28511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:17.893004 28511 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:35927
I20260811 03:04:17.893065 28653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:35927 every 8 connection(s)
I20260811 03:04:17.894464 28511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:04:17.899981 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 28511
I20260811 03:04:17.900189 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:04:17.900795 28084 external_mini_cluster.cc:1428] Setting key 264dd384ba9a2c38a9269022899cafec
Aug 11 03:04:17 dist-test-slave-8tdl krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417457, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:17.918068 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:17.897437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:36707 (local address 127.27.109.62:34339)
0811 03:04:17.897978 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:17.897983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:17.898962 (+ 979us) server_negotiation.cc:407] Connection header received
0811 03:04:17.901708 (+ 2746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:17.901724 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:17.901840 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:17.902015 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:17.903934 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.905052 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.906716 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.907132 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.907278 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:17.911588 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:17.911594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:17.911598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:17.911640 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:17.914362 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:17.915397 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:17.915402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:17.915404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:17.915546 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:17.916062 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:17.916066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:17.916068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:17.916405 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:17.916588 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:17.917408 (+ 820us) server_negotiation.cc:299] Negotiation successful
0811 03:04:17.917658 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":243,"threads_started":1}
I20260811 03:04:17.919169 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:17.897980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34339 (local address 127.27.109.3:36707)
0811 03:04:17.898776 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:17.898827 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:17.901296 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:17.902280 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:17.902288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:17.903031 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:17.903752 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:17.903772 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.905228 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.905232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:17.906514 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:17.906526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.907395 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.907402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:17.907653 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:17.908361 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:17.908392 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:17.911302 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:17.914587 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:17.914609 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:17.914626 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:17.915225 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:17.915678 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:17.915683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:17.915686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:17.915911 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:17.916549 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:17.916558 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:17.916974 (+ 416us) client_negotiation.cc:769] Sending connection context
0811 03:04:17.917278 (+ 304us) client_negotiation.cc:240] Negotiation successful
0811 03:04:17.917904 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":288,"threads_started":1}
I20260811 03:04:17.920706 28654 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34339
I20260811 03:04:17.921133 28654 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:17.921952 28654 heartbeater.cc:511] Master 127.27.109.62:34339 requested a full tablet report, sending...
I20260811 03:04:17.923832 28139 ts_manager.cc:194] Registered new tserver with Master: 1e132d9a63fd4d46914c407736bde419 (127.27.109.3:35927)
I20260811 03:04:17.924808 28139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:36707
I20260811 03:04:17.930606 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 03:04:17 dist-test-slave-8tdl krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417457, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 03:04:17 dist-test-slave-8tdl krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417457, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:17.986146 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:17.968790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.27.109.1:40499)
0811 03:04:17.969596 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:17.969607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:17.969656 (+ 49us) server_negotiation.cc:407] Connection header received
0811 03:04:17.969739 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:17.969746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:17.969994 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:17.970186 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:17.971861 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.972931 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.974238 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:17.974961 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:17.975111 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:17.979462 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:17.979477 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:17.979492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:17.979531 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:17.983030 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:17.983807 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:17.983815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:17.983817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:17.983959 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:17.984390 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:17.984395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:17.984398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:17.984789 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:17.985148 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:17.985443 (+ 295us) server_negotiation.cc:299] Negotiation successful
0811 03:04:17.985734 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":369,"threads_started":1}
W20260811 03:04:17.987684 28256 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49938
Aug 11 03:04:18 dist-test-slave-8tdl krb5kdc[28091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 03:04:18 dist-test-slave-8tdl krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:18.040896 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:18.024887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.27.109.1:40499)
0811 03:04:18.025160 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:18.025167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:18.025502 (+ 335us) server_negotiation.cc:407] Connection header received
0811 03:04:18.025751 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:18.025756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:18.025867 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:18.026006 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:18.027649 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:18.028424 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:18.029909 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:18.030471 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:18.030620 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:18.035361 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:18.035368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:18.035371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:18.035412 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:18.038145 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:18.038965 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:18.038969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:18.038971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:18.039156 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:18.039562 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:18.039565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:18.039567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:18.039833 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:18.040033 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:18.040365 (+ 332us) server_negotiation.cc:299] Negotiation successful
0811 03:04:18.040550 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 03:04:18.042518 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 28207
I20260811 03:04:18.057565 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:18.043651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.27.109.1:40499)
0811 03:04:18.043853 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:18.043862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:18.044236 (+ 374us) server_negotiation.cc:407] Connection header received
0811 03:04:18.044429 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:18.044434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:18.044501 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:18.044608 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:18.045863 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:18.046620 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:18.047986 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:18.048534 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:18.048675 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:18.051210 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:18.051214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:18.051218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:18.051252 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:18.054229 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:18.055229 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:18.055236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:18.055240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:18.055401 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:18.055836 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:18.055845 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:18.055849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:18.056247 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:18.056544 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:18.056873 (+ 329us) server_negotiation.cc:299] Negotiation successful
0811 03:04:18.057169 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:04:18.058406 28345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:18.148586 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 28207
I20260811 03:04:18.170934 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 28358
Aug 11 03:04:18 dist-test-slave-8tdl krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:04:18.188911 28672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:18.171952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.27.109.2:41269)
0811 03:04:18.172564 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:18.172571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:18.172598 (+ 27us) server_negotiation.cc:407] Connection header received
0811 03:04:18.172663 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:18.172670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:18.172905 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:18.173040 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:18.174230 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:18.175476 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:18.176857 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:18.177458 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:18.177572 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:18.182054 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:18.182069 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:18.182109 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:18.182154 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:18.185591 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:18.186448 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:18.186456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:18.186458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:18.186607 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:18.187132 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:18.187136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:18.187138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:18.187515 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:18.187766 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:18.188199 (+ 433us) server_negotiation.cc:299] Negotiation successful
0811 03:04:18.188428 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":330,"threads_started":1}
I20260811 03:04:18.190266 28498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:18.278452 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 28358
I20260811 03:04:18.299021 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 28511
Aug 11 03:04:18 dist-test-slave-8tdl krb5kdc[28091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:04:18.317157 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:18.300100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.27.109.3:35927)
0811 03:04:18.300737 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:18.300745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:18.300775 (+ 30us) server_negotiation.cc:407] Connection header received
0811 03:04:18.300839 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:18.300846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:18.301124 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:18.301282 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:18.302827 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:18.303916 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:18.305254 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:18.305839 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:18.305941 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:18.310663 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:18.310739 (+ 76us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:18.310764 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:18.310819 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:18.313874 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:18.314730 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:18.314739 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:18.314741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:18.314899 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:18.315333 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:18.315339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:18.315342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:18.315753 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:18.316094 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:18.316457 (+ 363us) server_negotiation.cc:299] Negotiation successful
0811 03:04:18.316776 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":307,"threads_started":1}
I20260811 03:04:18.318758 28649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:18.408326 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 28511
I20260811 03:04:18.429912 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 28105
I20260811 03:04:18.431490 28174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:18.544661 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 28105
2026-08-11T03:04:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4383 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:18 dist-test-slave-8tdl krb5kdc[28680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:18 dist-test-slave-8tdl krb5kdc[28680](info): set up 2 sockets
Aug 11 03:04:18 dist-test-slave-8tdl krb5kdc[28680](info): commencing operation
krb5kdc: starting...
W20260811 03:04:19.702932 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:19 dist-test-slave-8tdl krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:04:19.927027 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42461
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:42461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:20.282238 28697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:20.282583 28697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:20.282783 28697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:20.290562 28697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:20.290769 28697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:20.290835 28697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:20.290891 28697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:20.290932 28697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:20.301909 28697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:42461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:20.303428 28697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:20.304847 28697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:20.318010 28702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:20.318132 28706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:20.318051 28703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:20.318982 28697 server_base.cc:1034] running on GCE node
I20260811 03:04:20.319804 28697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:20.321342 28697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:20.322561 28697 hybrid_clock.cc:648] HybridClock initialized: now 1786417460322484 us; error 63 us; skew 500 ppm
I20260811 03:04:20.322959 28697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:20 dist-test-slave-8tdl krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417460, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:20.326632 28697 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:20.328399 28697 webserver.cc:460] Webserver started at http://127.27.109.62:37551/ using document root <none> and password file <none>
I20260811 03:04:20.329154 28697 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:20.329268 28697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:20.329564 28697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:20.333017 28697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "8efc299f57144c32978da871411752b5"
format_stamp: "Formatted at 2026-08-11 03:04:20 on dist-test-slave-8tdl"
server_key: "fcc36c2e5aaa1dd38d44749a02d451eb"
server_key_iv: "710f683e0e63c093c6972507d79fee66"
server_key_version: "encryptionkey@0"
I20260811 03:04:20.333814 28697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "8efc299f57144c32978da871411752b5"
format_stamp: "Formatted at 2026-08-11 03:04:20 on dist-test-slave-8tdl"
server_key: "fcc36c2e5aaa1dd38d44749a02d451eb"
server_key_iv: "710f683e0e63c093c6972507d79fee66"
server_key_version: "encryptionkey@0"
I20260811 03:04:20.339679 28697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 03:04:20.343598 28715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:20.345323 28697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:04:20.345572 28697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "8efc299f57144c32978da871411752b5"
format_stamp: "Formatted at 2026-08-11 03:04:20 on dist-test-slave-8tdl"
server_key: "fcc36c2e5aaa1dd38d44749a02d451eb"
server_key_iv: "710f683e0e63c093c6972507d79fee66"
server_key_version: "encryptionkey@0"
I20260811 03:04:20.345821 28697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:20.368754 28697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:20.374182 28697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:20.374555 28697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:20.397603 28697 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:42461
I20260811 03:04:20.397650 28773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:42461 every 8 connection(s)
I20260811 03:04:20.399271 28697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:20.400358 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 28697
I20260811 03:04:20.400656 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:20.401281 28084 external_mini_cluster.cc:1428] Setting key d6e94604708037f9a76e5eb028fe7bc1
I20260811 03:04:20.404587 28774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:20.420395 28774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5: Bootstrap starting.
Aug 11 03:04:20 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:20.424393 28774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:20.426064 28774 log.cc:824] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:20.429986 28774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5: No bootstrap required, opened a new log
I20260811 03:04:20.434031 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:20.403777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.27.109.62:42461)
0811 03:04:20.404674 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:20.404690 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:20.404744 (+ 54us) server_negotiation.cc:407] Connection header received
0811 03:04:20.407340 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:20.407374 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:20.407962 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:20.408506 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:20.410713 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:20.412873 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:20.415641 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:20.416861 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:20.417088 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:20.423402 (+ 6314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:20.423415 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:20.423436 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:20.423499 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:20.427786 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:20.428972 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:20.428985 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:20.428994 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:20.429228 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:20.429743 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:20.429748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:20.429751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:20.430596 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:20.430959 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:20.431580 (+ 621us) server_negotiation.cc:299] Negotiation successful
0811 03:04:20.432343 (+ 763us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":370,"threads_started":1}
I20260811 03:04:20.440752 28774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efc299f57144c32978da871411752b5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42461 } }
I20260811 03:04:20.441336 28774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:20.441465 28774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efc299f57144c32978da871411752b5, State: Initialized, Role: FOLLOWER
I20260811 03:04:20.442337 28774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efc299f57144c32978da871411752b5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42461 } }
I20260811 03:04:20.442601 28774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:20.442777 28774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:20.442971 28774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:20.445411 28774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efc299f57144c32978da871411752b5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42461 } }
I20260811 03:04:20.446036 28774 leader_election.cc:302] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8efc299f57144c32978da871411752b5; no voters:
I20260811 03:04:20.446938 28774 leader_election.cc:288] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:20.447096 28782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:20.448452 28782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [term 1 LEADER]: Becoming Leader. State: Replica: 8efc299f57144c32978da871411752b5, State: Running, Role: LEADER
I20260811 03:04:20.449215 28782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efc299f57144c32978da871411752b5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42461 } }
I20260811 03:04:20.450057 28774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:20.452984 28784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8efc299f57144c32978da871411752b5. Latest consensus state: current_term: 1 leader_uuid: "8efc299f57144c32978da871411752b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efc299f57144c32978da871411752b5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42461 } } }
I20260811 03:04:20.453507 28784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:20.454771 28791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:20.454434 28783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8efc299f57144c32978da871411752b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8efc299f57144c32978da871411752b5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42461 } } }
I20260811 03:04:20.454917 28783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5 [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:20.473057 28791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:20.487140 28791 catalog_manager.cc:1347] Generated new cluster ID: 395eb96ddb4d4a96a40db539fa775bcf
I20260811 03:04:20.487285 28791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:20.522084 28791 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:20.524478 28791 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:20.549659 28791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8efc299f57144c32978da871411752b5: Generated new TSK 0
I20260811 03:04:20.550923 28791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:04:20.622464 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42461
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:21.008715 28805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:21.009110 28805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:21.009295 28805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:21.017099 28805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:21.017270 28805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:21.017469 28805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:21.028162 28805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42461
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:21.029619 28805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:21.031070 28805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:21.045725 28813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:21.045804 28810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:21.045768 28811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:21.046830 28805 server_base.cc:1034] running on GCE node
I20260811 03:04:21.047711 28805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:21.048815 28805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:21.050081 28805 hybrid_clock.cc:648] HybridClock initialized: now 1786417461049989 us; error 68 us; skew 500 ppm
I20260811 03:04:21.050530 28805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:21 dist-test-slave-8tdl krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417461, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:21.054441 28805 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:04:21.056252 28805 webserver.cc:460] Webserver started at http://127.27.109.1:43981/ using document root <none> and password file <none>
I20260811 03:04:21.057067 28805 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:21.057178 28805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:21.057463 28805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:21.060704 28805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "a5d29741bbcb4dc19a9840fa34c87952"
format_stamp: "Formatted at 2026-08-11 03:04:21 on dist-test-slave-8tdl"
server_key: "64edf97b99f5c785c0305787ba94d8ec"
server_key_iv: "f4f02bf5b8ce9b6f5e216954953a891a"
server_key_version: "encryptionkey@0"
I20260811 03:04:21.061544 28805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "a5d29741bbcb4dc19a9840fa34c87952"
format_stamp: "Formatted at 2026-08-11 03:04:21 on dist-test-slave-8tdl"
server_key: "64edf97b99f5c785c0305787ba94d8ec"
server_key_iv: "f4f02bf5b8ce9b6f5e216954953a891a"
server_key_version: "encryptionkey@0"
I20260811 03:04:21.067279 28805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 03:04:21.071386 28823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:21.072959 28805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:04:21.073180 28805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "a5d29741bbcb4dc19a9840fa34c87952"
format_stamp: "Formatted at 2026-08-11 03:04:21 on dist-test-slave-8tdl"
server_key: "64edf97b99f5c785c0305787ba94d8ec"
server_key_iv: "f4f02bf5b8ce9b6f5e216954953a891a"
server_key_version: "encryptionkey@0"
I20260811 03:04:21.073450 28805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:21.106281 28805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:21.111014 28805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:21.111346 28805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:21.112460 28805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:21.114073 28805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:21.114187 28805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:21.114356 28805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:21.114441 28805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:21.151983 28805 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:40787
I20260811 03:04:21.152087 28952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:40787 every 8 connection(s)
I20260811 03:04:21.153501 28805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:21.158874 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 28805
I20260811 03:04:21.159125 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:21.159771 28084 external_mini_cluster.cc:1428] Setting key 4ec7d351b3dfedafea1a7dad90bef2c6
Aug 11 03:04:21 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417461, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:21.176934 28956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:21.156498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:40597 (local address 127.27.109.62:42461)
0811 03:04:21.157020 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:21.157025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:21.157866 (+ 841us) server_negotiation.cc:407] Connection header received
0811 03:04:21.160236 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:21.160251 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:21.160355 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:21.160494 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:21.162242 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:21.163181 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:21.165015 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:21.165461 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:21.165619 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:21.170410 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:21.170415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:21.170419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:21.170460 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:21.173464 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:21.174294 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:21.174300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:21.174304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:21.174455 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:21.174999 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:21.175004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:21.175007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:21.175355 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:21.175520 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:21.176283 (+ 763us) server_negotiation.cc:299] Negotiation successful
0811 03:04:21.176530 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":285,"threads_started":1}
I20260811 03:04:21.178102 28957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:21.156898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42461 (local address 127.27.109.1:40597)
0811 03:04:21.157664 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:21.157715 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:21.159895 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:21.160787 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:21.160797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:21.161402 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:21.162052 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:21.162072 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:21.163391 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:21.163395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:21.164830 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:21.164844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:21.165735 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:21.165745 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:21.166043 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:21.166902 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:21.166928 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:21.170109 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:21.173652 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:21.173662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:21.173682 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:21.174156 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:21.174577 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:21.174582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:21.174586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:21.174873 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:21.175495 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:21.175504 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:21.175889 (+ 385us) client_negotiation.cc:769] Sending connection context
0811 03:04:21.176177 (+ 288us) client_negotiation.cc:240] Negotiation successful
0811 03:04:21.176788 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260811 03:04:21.179982 28953 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42461
I20260811 03:04:21.180263 28953 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:21.181110 28953 heartbeater.cc:511] Master 127.27.109.62:42461 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260811 03:04:21.183769 28734 ts_manager.cc:194] Registered new tserver with Master: a5d29741bbcb4dc19a9840fa34c87952 (127.27.109.1:40787)
I20260811 03:04:21.185350 28734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:40597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:04:21.210286 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42461
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:21.579567 28960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:21.579952 28960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:21.580157 28960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:21.588093 28960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:21.588244 28960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:21.588442 28960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:04:21.599167 28960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42461
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:21.600648 28960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:21.602133 28960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:21.616998 28970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:21.616935 28968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:21.617034 28967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:21.618489 28960 server_base.cc:1034] running on GCE node
I20260811 03:04:21.619347 28960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:21.620436 28960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:21.621680 28960 hybrid_clock.cc:648] HybridClock initialized: now 1786417461621621 us; error 41 us; skew 500 ppm
I20260811 03:04:21.622125 28960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:21 dist-test-slave-8tdl krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417461, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:21.625864 28960 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:04:21.627717 28960 webserver.cc:460] Webserver started at http://127.27.109.2:46787/ using document root <none> and password file <none>
I20260811 03:04:21.628494 28960 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:21.628654 28960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:21.628991 28960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:21.632220 28960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "355bffe63ac0410a80cf79e6a30b7696"
format_stamp: "Formatted at 2026-08-11 03:04:21 on dist-test-slave-8tdl"
server_key: "35a3434893fe70a4336d80085a1d737e"
server_key_iv: "e1ca4199ad26884d89349f6c6bef04c8"
server_key_version: "encryptionkey@0"
I20260811 03:04:21.633111 28960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "355bffe63ac0410a80cf79e6a30b7696"
format_stamp: "Formatted at 2026-08-11 03:04:21 on dist-test-slave-8tdl"
server_key: "35a3434893fe70a4336d80085a1d737e"
server_key_iv: "e1ca4199ad26884d89349f6c6bef04c8"
server_key_version: "encryptionkey@0"
I20260811 03:04:21.639221 28960 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 03:04:21.643345 28977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:21.645030 28960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:04:21.645239 28960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "355bffe63ac0410a80cf79e6a30b7696"
format_stamp: "Formatted at 2026-08-11 03:04:21 on dist-test-slave-8tdl"
server_key: "35a3434893fe70a4336d80085a1d737e"
server_key_iv: "e1ca4199ad26884d89349f6c6bef04c8"
server_key_version: "encryptionkey@0"
I20260811 03:04:21.645468 28960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:21.670418 28960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:21.675560 28960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:21.675890 28960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:21.677062 28960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:21.678733 28960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:21.678845 28960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260811 03:04:21.679004 28960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:21.679095 28960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:21.715777 28960 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45133
I20260811 03:04:21.715876 29110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45133 every 8 connection(s)
I20260811 03:04:21.717176 28960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:04:21.726086 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 28960
I20260811 03:04:21.726310 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:04:21.726883 28084 external_mini_cluster.cc:1428] Setting key 1f896962b9d45a8e1947aa2270375954
Aug 11 03:04:21 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417461, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:21.746213 29113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:21.720212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:59695 (local address 127.27.109.62:42461)
0811 03:04:21.720716 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:21.720723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:21.722114 (+ 1391us) server_negotiation.cc:407] Connection header received
0811 03:04:21.725457 (+ 3343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:21.725464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:21.725570 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:21.725712 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:21.728990 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:21.730273 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:21.733751 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:21.734284 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:21.734404 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:21.739290 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:21.739296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:21.739300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:21.739340 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:21.742232 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:21.743441 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:21.743445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:21.743448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:21.743597 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:21.744220 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:21.744224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:21.744226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:21.744525 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:21.744645 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:21.745726 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0811 03:04:21.745921 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
I20260811 03:04:21.748054 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:21.720887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42461 (local address 127.27.109.2:59695)
0811 03:04:21.721894 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:21.721951 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:21.724991 (+ 3040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:21.726428 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:21.726455 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:21.727555 (+ 1100us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:21.728697 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:21.728733 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:21.730522 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:21.730530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:21.733503 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:21.733529 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:21.734578 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:21.734589 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:21.735021 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:21.735866 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:21.735904 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:21.738923 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:21.742438 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:21.742452 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:21.742476 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:21.743285 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:21.743731 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:21.743738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:21.743745 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:21.744082 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:21.744677 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:21.744689 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:21.745261 (+ 572us) client_negotiation.cc:769] Sending connection context
0811 03:04:21.745623 (+ 362us) client_negotiation.cc:240] Negotiation successful
0811 03:04:21.746346 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":365,"threads_started":1}
I20260811 03:04:21.749867 29111 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42461
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260811 03:04:21.750360 29111 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:21.751359 29111 heartbeater.cc:511] Master 127.27.109.62:42461 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260811 03:04:21.753278 28734 ts_manager.cc:194] Registered new tserver with Master: 355bffe63ac0410a80cf79e6a30b7696 (127.27.109.2:45133)
I20260811 03:04:21.754361 28734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:59695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:04:21.782004 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42461
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:22.161013 29117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:22.161334 29117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:22.161438 29117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:22.169395 29117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:22.169545 29117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:22.169669 29117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:04:22.180732 29117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42461
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:22.182161 29117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:22.183952 29117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 03:04:22.189692 28953 heartbeater.cc:503] Master 127.27.109.62:42461 was elected leader, sending a full tablet report...
W20260811 03:04:22.200520 29124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:22.200521 29125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:22.201053 29127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:22.201263 29117 server_base.cc:1034] running on GCE node
I20260811 03:04:22.202127 29117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:22.203385 29117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:22.204636 29117 hybrid_clock.cc:648] HybridClock initialized: now 1786417462204582 us; error 38 us; skew 500 ppm
I20260811 03:04:22.205060 29117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:22 dist-test-slave-8tdl krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:22.209036 29117 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:04:22.210997 29117 webserver.cc:460] Webserver started at http://127.27.109.3:41237/ using document root <none> and password file <none>
I20260811 03:04:22.211890 29117 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:22.212023 29117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:22.212320 29117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:22.215983 29117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "f28121409f7147cfa637b3e5bec16c15"
format_stamp: "Formatted at 2026-08-11 03:04:22 on dist-test-slave-8tdl"
server_key: "c6afb9353277ffc3641cb1cad8e2fe8f"
server_key_iv: "feafc2328e8b5890ee68b4880a1d5016"
server_key_version: "encryptionkey@0"
I20260811 03:04:22.216920 29117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "f28121409f7147cfa637b3e5bec16c15"
format_stamp: "Formatted at 2026-08-11 03:04:22 on dist-test-slave-8tdl"
server_key: "c6afb9353277ffc3641cb1cad8e2fe8f"
server_key_iv: "feafc2328e8b5890ee68b4880a1d5016"
server_key_version: "encryptionkey@0"
I20260811 03:04:22.223552 29117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260811 03:04:22.229125 29135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:22.230780 29117 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 03:04:22.231069 29117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "f28121409f7147cfa637b3e5bec16c15"
format_stamp: "Formatted at 2026-08-11 03:04:22 on dist-test-slave-8tdl"
server_key: "c6afb9353277ffc3641cb1cad8e2fe8f"
server_key_iv: "feafc2328e8b5890ee68b4880a1d5016"
server_key_version: "encryptionkey@0"
I20260811 03:04:22.231343 29117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:22.253815 29117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:22.259223 29117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:22.259588 29117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:22.260718 29117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:22.262360 29117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:22.262490 29117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:22.262662 29117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:22.262832 29117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260811 03:04:22.302971 29117 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:43225
I20260811 03:04:22.303004 29262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:43225 every 8 connection(s)
I20260811 03:04:22.304489 29117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:04:22.306624 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 29117
I20260811 03:04:22.306879 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:04:22.307389 28084 external_mini_cluster.cc:1428] Setting key ec85931f185dd5e94e369be0f2c8d4a5
Aug 11 03:04:22 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:22.330085 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.307900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:47695 (local address 127.27.109.62:42461)
0811 03:04:22.308414 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.308419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.309937 (+ 1518us) server_negotiation.cc:407] Connection header received
0811 03:04:22.312503 (+ 2566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.312508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.312586 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.312695 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:22.314460 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.315470 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.317650 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.318116 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.318245 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.323087 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:22.323102 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:22.323106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:22.323162 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:22.326635 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.327681 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.327685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.327688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.327842 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.328289 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.328293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.328295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.328592 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:22.328716 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.329578 (+ 862us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.329776 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
I20260811 03:04:22.331272 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.308648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42461 (local address 127.27.109.3:47695)
0811 03:04:22.309748 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:22.309812 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:22.312009 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:22.312951 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:22.312961 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:22.313601 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:22.314265 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.314285 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.315664 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.315669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.317478 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.317494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.318397 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.318403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.318760 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.319589 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:22.319618 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:22.322603 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:22.326902 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:22.326910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:22.326927 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:22.327548 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:22.327953 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:22.327956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:22.327959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:22.328182 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:22.328753 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:22.328760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:22.329195 (+ 435us) client_negotiation.cc:769] Sending connection context
0811 03:04:22.329461 (+ 266us) client_negotiation.cc:240] Negotiation successful
0811 03:04:22.329918 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":454,"threads_started":1}
I20260811 03:04:22.332949 29263 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42461
I20260811 03:04:22.333251 29263 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:22.334014 29263 heartbeater.cc:511] Master 127.27.109.62:42461 requested a full tablet report, sending...
I20260811 03:04:22.336107 28734 ts_manager.cc:194] Registered new tserver with Master: f28121409f7147cfa637b3e5bec16c15 (127.27.109.3:43225)
I20260811 03:04:22.337208 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:04:22.337342 28734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:47695
I20260811 03:04:22.360064 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.344337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.27.109.62:42461)
0811 03:04:22.344554 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.344560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.345189 (+ 629us) server_negotiation.cc:407] Connection header received
0811 03:04:22.345330 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.345334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.345410 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.345517 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:22.347400 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.348293 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.349940 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.350603 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.350840 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.353929 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:22.353936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:22.353939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:22.353984 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:22.357041 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.357788 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.357792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.357795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.358005 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.358416 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.358421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.358425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.358884 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:22.359157 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.359491 (+ 334us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.359695 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 03:04:22.367522 28734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:04:22.375684 28734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:22.403679 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.390344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:40787 (local address 127.0.0.1:56448)
0811 03:04:22.391302 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:22.391351 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:22.391550 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:22.394062 (+ 2512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:22.394066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:22.394088 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:22.394632 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.394651 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.397532 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.397537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.399970 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.399994 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.402345 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.402352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.402623 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.402736 (+ 113us) client_negotiation.cc:769] Sending connection context
0811 03:04:22.402891 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 03:04:22.403157 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":71,"thread_start_us":1792,"threads_started":1}
I20260811 03:04:22.404858 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.393305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45133 (local address 127.0.0.1:45036)
0811 03:04:22.395721 (+ 2416us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:22.395739 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:22.395864 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:22.396727 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:22.396732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:22.396756 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:22.397266 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.397278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.399334 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.399340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.401720 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.401734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.403947 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.403953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.404139 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.404173 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:04:22.404258 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 03:04:22.404400 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2279,"mutex_wait_us":3,"thread_start_us":2142,"threads_started":1}
I20260811 03:04:22.405072 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.390779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.27.109.1:40787)
0811 03:04:22.393380 (+ 2601us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.393390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.393423 (+ 33us) server_negotiation.cc:407] Connection header received
0811 03:04:22.393546 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.393554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.393833 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.394484 (+ 651us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:22.395379 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.397337 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.400268 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.402074 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.402549 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.404313 (+ 1764us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.404425 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.404672 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2339,"thread_start_us":2139,"threads_started":1}
I20260811 03:04:22.406055 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.392889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.27.109.2:45133)
0811 03:04:22.395261 (+ 2372us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.395273 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.396003 (+ 730us) server_negotiation.cc:407] Connection header received
0811 03:04:22.396123 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.396130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.396375 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.396583 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:22.397479 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.399152 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.401922 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.403441 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.403602 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.403804 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.405499 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.405712 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2177,"thread_start_us":2078,"threads_started":1}
I20260811 03:04:22.408552 29278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.390464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:43225 (local address 127.0.0.1:33800)
0811 03:04:22.392486 (+ 2022us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:22.392507 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:22.392653 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:22.393991 (+ 1338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:22.394013 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:22.394063 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:22.395217 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.395232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.397524 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.397530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.404839 (+ 7309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.404860 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.407608 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.407615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.407825 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.407870 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 03:04:22.407961 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 03:04:22.408125 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1879,"mutex_wait_us":56,"thread_start_us":380,"threads_started":1}
I20260811 03:04:22.409701 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.391293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.27.109.3:43225)
0811 03:04:22.392777 (+ 1484us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.392789 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.392833 (+ 44us) server_negotiation.cc:407] Connection header received
0811 03:04:22.392960 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.392970 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.393297 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.393582 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:22.395477 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.397351 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.405119 (+ 7768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.406974 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.407140 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.407405 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.409129 (+ 1724us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.409325 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":1050,"threads_started":1}
I20260811 03:04:22.409790 29035 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef69bebdc3e4fc89c364d886268a9cc (DEFAULT_TABLE table=test-table [id=46b63fbf94b5490d8ba302f0bb2c0efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:22.409826 28876 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef69bebdc3e4fc89c364d886268a9cc (DEFAULT_TABLE table=test-table [id=46b63fbf94b5490d8ba302f0bb2c0efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:22.411689 28876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef69bebdc3e4fc89c364d886268a9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:22.413671 29189 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef69bebdc3e4fc89c364d886268a9cc (DEFAULT_TABLE table=test-table [id=46b63fbf94b5490d8ba302f0bb2c0efc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:22.415323 29189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef69bebdc3e4fc89c364d886268a9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:22.415798 29035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef69bebdc3e4fc89c364d886268a9cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:22.428335 29284 tablet_bootstrap.cc:492] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952: Bootstrap starting.
I20260811 03:04:22.429350 29285 tablet_bootstrap.cc:492] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15: Bootstrap starting.
I20260811 03:04:22.432893 29284 tablet_bootstrap.cc:654] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:22.433485 29287 tablet_bootstrap.cc:492] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696: Bootstrap starting.
I20260811 03:04:22.433593 29285 tablet_bootstrap.cc:654] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:22.434975 29285 log.cc:824] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:22.435012 29284 log.cc:824] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:22.437314 29287 tablet_bootstrap.cc:654] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:22.438731 29285 tablet_bootstrap.cc:492] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15: No bootstrap required, opened a new log
I20260811 03:04:22.438789 29284 tablet_bootstrap.cc:492] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952: No bootstrap required, opened a new log
I20260811 03:04:22.439034 29285 ts_tablet_manager.cc:1397] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260811 03:04:22.439118 29284 ts_tablet_manager.cc:1397] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260811 03:04:22.439684 29287 log.cc:824] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:22.443375 29287 tablet_bootstrap.cc:492] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696: No bootstrap required, opened a new log
I20260811 03:04:22.443712 29287 ts_tablet_manager.cc:1397] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260811 03:04:22.448769 29284 raft_consensus.cc:348] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.449440 29284 raft_consensus.cc:374] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:22.449582 29284 raft_consensus.cc:729] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d29741bbcb4dc19a9840fa34c87952, State: Initialized, Role: FOLLOWER
I20260811 03:04:22.450145 29285 raft_consensus.cc:348] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.450739 29285 raft_consensus.cc:374] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:22.450846 29285 raft_consensus.cc:729] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28121409f7147cfa637b3e5bec16c15, State: Initialized, Role: FOLLOWER
I20260811 03:04:22.450613 29284 consensus_queue.cc:260] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.451627 29285 consensus_queue.cc:260] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.453104 29263 heartbeater.cc:503] Master 127.27.109.62:42461 was elected leader, sending a full tablet report...
I20260811 03:04:22.453799 29285 ts_tablet_manager.cc:1428] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260811 03:04:22.455699 29284 ts_tablet_manager.cc:1428] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260811 03:04:22.456130 29287 raft_consensus.cc:348] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.456614 29287 raft_consensus.cc:374] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:22.456734 29287 raft_consensus.cc:729] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355bffe63ac0410a80cf79e6a30b7696, State: Initialized, Role: FOLLOWER
I20260811 03:04:22.457286 29287 consensus_queue.cc:260] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.460456 29111 heartbeater.cc:503] Master 127.27.109.62:42461 was elected leader, sending a full tablet report...
I20260811 03:04:22.461319 29287 ts_tablet_manager.cc:1428] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260811 03:04:22.469375 29112 tablet.cc:2369] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:04:22.556638 29264 tablet.cc:2369] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:04:22.560374 29293 raft_consensus.cc:482] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:22.560757 29293 raft_consensus.cc:504] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.562903 29293 leader_election.cc:288] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5d29741bbcb4dc19a9840fa34c87952 (127.27.109.1:40787), 355bffe63ac0410a80cf79e6a30b7696 (127.27.109.2:45133)
I20260811 03:04:22.567104 29292 raft_consensus.cc:482] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:22.567487 29292 raft_consensus.cc:504] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.569854 29292 leader_election.cc:288] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f28121409f7147cfa637b3e5bec16c15 (127.27.109.3:43225), 355bffe63ac0410a80cf79e6a30b7696 (127.27.109.2:45133)
I20260811 03:04:22.571558 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.563394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:48143 (local address 127.27.109.1:40787)
0811 03:04:22.563689 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.563696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.563997 (+ 301us) server_negotiation.cc:407] Connection header received
0811 03:04:22.564093 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.564098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.564191 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.564286 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:22.564989 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.566194 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.568572 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.569530 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.569714 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.569801 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.571115 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.571275 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260811 03:04:22.572124 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.563379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:40787 (local address 127.27.109.3:48143)
0811 03:04:22.563661 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:22.563685 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:22.563847 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:22.564511 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:22.564516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:22.564543 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:22.564760 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.564772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.566344 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.566351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.568366 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.568380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.569708 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.569714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.570567 (+ 853us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.570620 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 03:04:22.571734 (+ 1114us) client_negotiation.cc:240] Negotiation successful
0811 03:04:22.571872 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":3}
I20260811 03:04:22.572646 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.564197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:51795 (local address 127.27.109.2:45133)
0811 03:04:22.564431 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.564438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.564752 (+ 314us) server_negotiation.cc:407] Connection header received
0811 03:04:22.564912 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.564920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.565010 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.565089 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:22.565830 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.566954 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.569547 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.570961 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.571134 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.571216 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.571571 (+ 355us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.572054 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 03:04:22.572132 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.563820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45133 (local address 127.27.109.3:51795)
0811 03:04:22.564627 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:22.564647 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:22.564769 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:22.565202 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:22.565210 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:22.565239 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:22.565657 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.565673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.567155 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.567161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.569346 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.569361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.571232 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.571239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.571398 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.571432 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:04:22.571534 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 03:04:22.571688 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":27,"thread_start_us":492,"threads_started":1}
I20260811 03:04:22.573819 28900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef69bebdc3e4fc89c364d886268a9cc" candidate_uuid: "f28121409f7147cfa637b3e5bec16c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d29741bbcb4dc19a9840fa34c87952" is_pre_election: true
I20260811 03:04:22.574503 28900 raft_consensus.cc:2454] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f28121409f7147cfa637b3e5bec16c15 in term 0.
I20260811 03:04:22.575449 29139 leader_election.cc:302] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d29741bbcb4dc19a9840fa34c87952, f28121409f7147cfa637b3e5bec16c15; no voters:
I20260811 03:04:22.575474 29058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef69bebdc3e4fc89c364d886268a9cc" candidate_uuid: "f28121409f7147cfa637b3e5bec16c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355bffe63ac0410a80cf79e6a30b7696" is_pre_election: true
I20260811 03:04:22.576074 29293 raft_consensus.cc:2781] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:22.576109 29058 raft_consensus.cc:2454] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f28121409f7147cfa637b3e5bec16c15 in term 0.
I20260811 03:04:22.576249 29293 raft_consensus.cc:482] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:22.576380 29293 raft_consensus.cc:3037] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:22.580261 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.571366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44271 (local address 127.27.109.2:45133)
0811 03:04:22.572937 (+ 1571us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.572943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.572967 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:04:22.573039 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.573045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.573124 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.573234 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:22.574087 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.575365 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.578013 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.579125 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.579324 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.579403 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.579755 (+ 352us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.579907 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421,"thread_start_us":1093,"threads_started":1}
I20260811 03:04:22.580328 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.570161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45133 (local address 127.27.109.1:44271)
0811 03:04:22.571202 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:22.571225 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:22.571367 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:22.573538 (+ 2171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:22.573543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:22.573570 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:22.573911 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.573920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.575558 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.575564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.577807 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.577821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.579354 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.579361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.579562 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.579602 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 03:04:22.579709 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 03:04:22.579854 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":2669,"threads_started":1}
I20260811 03:04:22.580338 29293 raft_consensus.cc:504] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.581239 29058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef69bebdc3e4fc89c364d886268a9cc" candidate_uuid: "a5d29741bbcb4dc19a9840fa34c87952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355bffe63ac0410a80cf79e6a30b7696" is_pre_election: true
I20260811 03:04:22.581522 29058 raft_consensus.cc:2454] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d29741bbcb4dc19a9840fa34c87952 in term 0.
I20260811 03:04:22.581631 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.570439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:43225 (local address 127.27.109.1:50415)
0811 03:04:22.573098 (+ 2659us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:22.573121 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:22.573324 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:22.573915 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:22.573919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:22.573947 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:22.574466 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.574483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.576149 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.576154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.578596 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:22.578615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.580795 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.580801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:22.580989 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.581039 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 03:04:22.581146 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 03:04:22.581295 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2516,"thread_start_us":381,"threads_started":1}
I20260811 03:04:22.581642 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.570408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50415 (local address 127.27.109.3:43225)
0811 03:04:22.570632 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.570639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.573210 (+ 2571us) server_negotiation.cc:407] Connection header received
0811 03:04:22.573518 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.573523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.573608 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.573682 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:22.574744 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.575929 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.578793 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.580207 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.580404 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.580490 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.581198 (+ 708us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.581363 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 03:04:22.581736 28900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef69bebdc3e4fc89c364d886268a9cc" candidate_uuid: "f28121409f7147cfa637b3e5bec16c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d29741bbcb4dc19a9840fa34c87952"
I20260811 03:04:22.582005 28900 raft_consensus.cc:3037] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:22.582129 28827 leader_election.cc:302] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355bffe63ac0410a80cf79e6a30b7696, a5d29741bbcb4dc19a9840fa34c87952; no voters:
I20260811 03:04:22.582996 29293 leader_election.cc:288] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [CANDIDATE]: Term 1 election: Requested vote from peers a5d29741bbcb4dc19a9840fa34c87952 (127.27.109.1:40787), 355bffe63ac0410a80cf79e6a30b7696 (127.27.109.2:45133)
I20260811 03:04:22.583498 29212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef69bebdc3e4fc89c364d886268a9cc" candidate_uuid: "a5d29741bbcb4dc19a9840fa34c87952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28121409f7147cfa637b3e5bec16c15" is_pre_election: true
I20260811 03:04:22.583666 29058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef69bebdc3e4fc89c364d886268a9cc" candidate_uuid: "f28121409f7147cfa637b3e5bec16c15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355bffe63ac0410a80cf79e6a30b7696"
I20260811 03:04:22.583902 29058 raft_consensus.cc:3037] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:22.584034 29212 raft_consensus.cc:2379] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5d29741bbcb4dc19a9840fa34c87952 in current term 1: Already voted for candidate f28121409f7147cfa637b3e5bec16c15 in this term.
I20260811 03:04:22.585866 28900 raft_consensus.cc:2454] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f28121409f7147cfa637b3e5bec16c15 in term 1.
I20260811 03:04:22.586169 29292 raft_consensus.cc:2741] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260811 03:04:22.586369 29139 leader_election.cc:302] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5d29741bbcb4dc19a9840fa34c87952, f28121409f7147cfa637b3e5bec16c15; no voters:
I20260811 03:04:22.586746 29293 raft_consensus.cc:2781] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:22.587409 29293 raft_consensus.cc:686] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [term 1 LEADER]: Becoming Leader. State: Replica: f28121409f7147cfa637b3e5bec16c15, State: Running, Role: LEADER
I20260811 03:04:22.587790 29058 raft_consensus.cc:2454] T 2ef69bebdc3e4fc89c364d886268a9cc P 355bffe63ac0410a80cf79e6a30b7696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f28121409f7147cfa637b3e5bec16c15 in term 1.
I20260811 03:04:22.588070 29293 consensus_queue.cc:237] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } }
I20260811 03:04:22.593467 28734 catalog_manager.cc:5495] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 reported cstate change: term changed from 0 to 1, leader changed from <none> to f28121409f7147cfa637b3e5bec16c15 (127.27.109.3). New cstate: current_term: 1 leader_uuid: "f28121409f7147cfa637b3e5bec16c15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5d29741bbcb4dc19a9840fa34c87952" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28121409f7147cfa637b3e5bec16c15" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 43225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "355bffe63ac0410a80cf79e6a30b7696" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45133 } health_report { overall_health: UNKNOWN } } }
Aug 11 03:04:22 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:22.619622 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.603303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.27.109.1:40787)
0811 03:04:22.603510 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.603515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.604085 (+ 570us) server_negotiation.cc:407] Connection header received
0811 03:04:22.604244 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.604248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.604311 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.604397 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:22.606088 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.606914 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.608272 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.608723 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.608828 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.613467 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:22.613502 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:22.613519 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:22.613565 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:22.616747 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.617535 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.617543 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.617545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.617694 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.618075 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.618079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.618081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.618502 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:22.618804 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.619084 (+ 280us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.619256 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Aug 11 03:04:22 dist-test-slave-8tdl krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260811 03:04:22.656769 28955 tablet.cc:2369] T 2ef69bebdc3e4fc89c364d886268a9cc P a5d29741bbcb4dc19a9840fa34c87952: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 03:04:22 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:22.677708 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.662811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.27.109.1:40787)
0811 03:04:22.663054 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.663065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.663457 (+ 392us) server_negotiation.cc:407] Connection header received
0811 03:04:22.663620 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.663624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.663691 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.663778 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:22.665173 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.666065 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.667465 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.667939 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.668052 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.672110 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:22.672115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:22.672119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:22.672158 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:22.675144 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.675770 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.675774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.675776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.675970 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.676350 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.676354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.676356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.676639 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:22.676841 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.677105 (+ 264us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.677334 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 11 03:04:22 dist-test-slave-8tdl krb5kdc[28680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 03:04:22 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:22.733976 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.716294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.27.109.1:40787)
0811 03:04:22.716516 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.716524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.716913 (+ 389us) server_negotiation.cc:407] Connection header received
0811 03:04:22.717057 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.717064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.717164 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.717298 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:22.719093 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.720325 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.721923 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.722714 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.722862 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.727133 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:22.727140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:22.727144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:22.727194 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:22.730804 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.731643 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.731648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.731650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.731779 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.732296 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.732300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.732303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.732701 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:22.733019 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.733300 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.733503 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260811 03:04:22.736300 28854 tablet_service.cc:2908] Rejecting scan request for tablet 2ef69bebdc3e4fc89c364d886268a9cc: Uninitialized: clean time has not yet been initialized
I20260811 03:04:22.740437 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 28805
I20260811 03:04:22.755527 29280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.741443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.27.109.1:40787)
0811 03:04:22.741690 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.741700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.742010 (+ 310us) server_negotiation.cc:407] Connection header received
0811 03:04:22.742219 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.742223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.742304 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.742397 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:22.743772 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.744503 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.745713 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.746239 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.746362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.748855 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:22.748861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:22.748866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:22.748908 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:22.752533 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.753324 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.753329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.753331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.753553 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.753959 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.753962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.753964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.754304 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:22.754573 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.754883 (+ 310us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.755147 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 03:04:22.756222 28946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:22.853601 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 28805
I20260811 03:04:22.875975 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 28960
Aug 11 03:04:22 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:04:22.892059 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:22.876785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.27.109.2:45133)
0811 03:04:22.877034 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:22.877039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:22.877058 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:04:22.877120 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:22.877124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:22.877188 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:22.877279 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:22.878486 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.879475 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.880775 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:22.881271 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:22.881396 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:22.885730 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:22.885747 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:22.885765 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:22.885811 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:22.889106 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.889836 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.889844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.889846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.890017 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:22.890401 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:22.890405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:22.890408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:22.890911 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:22.891159 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:22.891462 (+ 303us) server_negotiation.cc:299] Negotiation successful
0811 03:04:22.891660 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 03:04:22.892948 29106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:22.986656 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 28960
W20260811 03:04:23.008494 29139 connection.cc:535] client connection to 127.27.109.2:45133 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 03:04:23.008827 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 29117
W20260811 03:04:23.008831 29139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 03:04:23.010752 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:23.010221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45133 (local address 127.27.109.3:43699)
0811 03:04:23.010413 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:23.010536 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.2:45133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260811 03:04:23.011055 29139 consensus_peers.cc:487] T 2ef69bebdc3e4fc89c364d886268a9cc P f28121409f7147cfa637b3e5bec16c15 -> Peer 355bffe63ac0410a80cf79e6a30b7696 (127.27.109.2:45133): Couldn't send request to peer 355bffe63ac0410a80cf79e6a30b7696. Status: Network error: Client connection negotiation failed: client connection to 127.27.109.2:45133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Aug 11 03:04:23 dist-test-slave-8tdl krb5kdc[28680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:04:23.024817 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:23.009698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.27.109.3:43225)
0811 03:04:23.009867 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:23.009872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:23.009927 (+ 55us) server_negotiation.cc:407] Connection header received
0811 03:04:23.010193 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:23.010198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:23.010286 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:23.010395 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:23.011862 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:23.012659 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:23.013952 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:23.014344 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:23.014440 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:23.019157 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:23.019171 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:23.019195 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:23.019234 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:23.022021 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:23.022777 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:23.022789 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:23.022791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:23.022934 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:23.023305 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:23.023308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:23.023309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:23.023702 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:23.023929 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:23.024257 (+ 328us) server_negotiation.cc:299] Negotiation successful
0811 03:04:23.024439 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 03:04:23.025547 29258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:23.120374 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 29117
I20260811 03:04:23.142043 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 28697
I20260811 03:04:23.143365 28768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:23.257320 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 28697
2026-08-11T03:04:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4712 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:23 dist-test-slave-8tdl krb5kdc[29328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:23 dist-test-slave-8tdl krb5kdc[29328](info): set up 2 sockets
Aug 11 03:04:23 dist-test-slave-8tdl krb5kdc[29328](info): commencing operation
krb5kdc: starting...
W20260811 03:04:24.449888 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:24 dist-test-slave-8tdl krb5kdc[29328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:04:24.679155 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39885
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:25.028553 29342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:25.028940 29342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:25.029116 29342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:25.036692 29342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:25.036878 29342 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 03:04:25.036981 29342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:25.037103 29342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:25.037202 29342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:25.037297 29342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:25.047879 29342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39885
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:25.049311 29342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:25.050804 29342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:25.063841 29347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:25.063777 29348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:25.063799 29350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:25.065306 29342 server_base.cc:1034] running on GCE node
I20260811 03:04:25.066195 29342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:25.067823 29342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:25.069069 29342 hybrid_clock.cc:648] HybridClock initialized: now 1786417465068980 us; error 64 us; skew 500 ppm
I20260811 03:04:25.069417 29342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:25 dist-test-slave-8tdl krb5kdc[29328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:25.072989 29342 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:25.074792 29342 webserver.cc:460] Webserver started at http://127.27.109.62:36941/ using document root <none> and password file <none>
I20260811 03:04:25.075603 29342 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:25.075727 29342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:25.076082 29342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:25.079453 29342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "5b363d676ce1419ab56e12332da6e7dd"
format_stamp: "Formatted at 2026-08-11 03:04:25 on dist-test-slave-8tdl"
server_key: "05c7bbe5d9d3c10e48b904c3ca82eedf"
server_key_iv: "f9fa9e1e4cae90c51f6fc3ac4c9c1d80"
server_key_version: "encryptionkey@0"
I20260811 03:04:25.080307 29342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "5b363d676ce1419ab56e12332da6e7dd"
format_stamp: "Formatted at 2026-08-11 03:04:25 on dist-test-slave-8tdl"
server_key: "05c7bbe5d9d3c10e48b904c3ca82eedf"
server_key_iv: "f9fa9e1e4cae90c51f6fc3ac4c9c1d80"
server_key_version: "encryptionkey@0"
I20260811 03:04:25.086251 29342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:04:25.090307 29358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:25.092029 29342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:04:25.092274 29342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "5b363d676ce1419ab56e12332da6e7dd"
format_stamp: "Formatted at 2026-08-11 03:04:25 on dist-test-slave-8tdl"
server_key: "05c7bbe5d9d3c10e48b904c3ca82eedf"
server_key_iv: "f9fa9e1e4cae90c51f6fc3ac4c9c1d80"
server_key_version: "encryptionkey@0"
I20260811 03:04:25.092563 29342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:25.153501 29342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:25.158098 29342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:25.158417 29342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:25.183000 29342 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39885
I20260811 03:04:25.183054 29428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39885 every 8 connection(s)
I20260811 03:04:25.184571 29342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:25.188637 29429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:25.193061 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 29342
I20260811 03:04:25.193328 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:25.194032 28084 external_mini_cluster.cc:1428] Setting key 2fed91cff3f9eb2462932ee9e0a8c4f5
I20260811 03:04:25.201004 29429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd: Bootstrap starting.
I20260811 03:04:25.204555 29429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:25.205744 29429 log.cc:824] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:25.209522 29429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd: No bootstrap required, opened a new log
Aug 11 03:04:25 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:25.218667 29429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b363d676ce1419ab56e12332da6e7dd" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39885 } }
I20260811 03:04:25.219163 29429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:25.219293 29429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b363d676ce1419ab56e12332da6e7dd, State: Initialized, Role: FOLLOWER
I20260811 03:04:25.219686 29436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:25.196514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.27.109.62:39885)
0811 03:04:25.197557 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:25.197569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:25.197679 (+ 110us) server_negotiation.cc:407] Connection header received
0811 03:04:25.199769 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:25.199796 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:25.200267 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:25.200694 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:25.202279 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.203764 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.205139 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.205869 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.206062 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:25.211429 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:25.211438 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:25.211456 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:25.211505 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:25.215086 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:25.215877 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:25.215885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:25.215892 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:25.216095 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:25.216501 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:25.216519 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:25.216522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:25.217134 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:25.217481 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:25.217982 (+ 501us) server_negotiation.cc:299] Negotiation successful
0811 03:04:25.218432 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":530,"threads_started":1}
I20260811 03:04:25.220050 29429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b363d676ce1419ab56e12332da6e7dd" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39885 } }
I20260811 03:04:25.220324 29429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:25.220491 29429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:25.220690 29429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:25.223294 29429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b363d676ce1419ab56e12332da6e7dd" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39885 } }
I20260811 03:04:25.223836 29429 leader_election.cc:302] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b363d676ce1419ab56e12332da6e7dd; no voters:
I20260811 03:04:25.224450 29429 leader_election.cc:288] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:25.224677 29438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:25.225526 29438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [term 1 LEADER]: Becoming Leader. State: Replica: 5b363d676ce1419ab56e12332da6e7dd, State: Running, Role: LEADER
I20260811 03:04:25.226063 29438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b363d676ce1419ab56e12332da6e7dd" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39885 } }
I20260811 03:04:25.226764 29429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:25.229226 29439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b363d676ce1419ab56e12332da6e7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b363d676ce1419ab56e12332da6e7dd" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39885 } } }
I20260811 03:04:25.229594 29439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:25.230489 29446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:25.230345 29440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b363d676ce1419ab56e12332da6e7dd. Latest consensus state: current_term: 1 leader_uuid: "5b363d676ce1419ab56e12332da6e7dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b363d676ce1419ab56e12332da6e7dd" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39885 } } }
I20260811 03:04:25.230695 29440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:25.236634 29446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:25.247818 29446 catalog_manager.cc:1347] Generated new cluster ID: 3a8e979bbfc74574b5c42f27c8678614
I20260811 03:04:25.247967 29446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:25.267570 29446 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:25.269025 29446 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:25.277938 29446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd: Generated new TSK 0
I20260811 03:04:25.278971 29446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:04:25.343075 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39885
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:25.726799 29461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:25.727432 29461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:25.727625 29461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:25.737113 29461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:25.737274 29461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:25.737421 29461 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:04:25.737520 29461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:25.748045 29461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:25.749399 29461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:25.750793 29461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:25.765031 29466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:25.765134 29469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:25.765096 29467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:25.766129 29461 server_base.cc:1034] running on GCE node
I20260811 03:04:25.766933 29461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:25.768014 29461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:25.769253 29461 hybrid_clock.cc:648] HybridClock initialized: now 1786417465769174 us; error 44 us; skew 500 ppm
I20260811 03:04:25.769620 29461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:25 dist-test-slave-8tdl krb5kdc[29328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:25.773018 29461 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:04:25.774631 29461 webserver.cc:460] Webserver started at http://127.27.109.1:41341/ using document root <none> and password file <none>
I20260811 03:04:25.775434 29461 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:25.775525 29461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:25.775795 29461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:25.779009 29461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "bcfe39d86d56402e9327f5329c612a61"
format_stamp: "Formatted at 2026-08-11 03:04:25 on dist-test-slave-8tdl"
server_key: "4c25e5230e0ef9233374813cee2d246d"
server_key_iv: "421cd6053070f9ea5bd44fd0cb3db618"
server_key_version: "encryptionkey@0"
I20260811 03:04:25.779815 29461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "bcfe39d86d56402e9327f5329c612a61"
format_stamp: "Formatted at 2026-08-11 03:04:25 on dist-test-slave-8tdl"
server_key: "4c25e5230e0ef9233374813cee2d246d"
server_key_iv: "421cd6053070f9ea5bd44fd0cb3db618"
server_key_version: "encryptionkey@0"
I20260811 03:04:25.785589 29461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:04:25.789844 29477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:25.791625 29461 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:04:25.791839 29461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "bcfe39d86d56402e9327f5329c612a61"
format_stamp: "Formatted at 2026-08-11 03:04:25 on dist-test-slave-8tdl"
server_key: "4c25e5230e0ef9233374813cee2d246d"
server_key_iv: "421cd6053070f9ea5bd44fd0cb3db618"
server_key_version: "encryptionkey@0"
I20260811 03:04:25.792121 29461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:25.811450 29461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:25.816303 29461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:25.816617 29461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:25.820056 29461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:25.820200 29461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:25.820397 29461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:25.820487 29461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:04:25 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:25.850055 29489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:25.821203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39885 (local address 127.27.109.1:57613)
0811 03:04:25.822263 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:25.822326 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:25.824574 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:25.825473 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:25.825489 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:25.826539 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:25.827818 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:25.827857 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.829331 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.829337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:25.832269 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:25.832298 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.833342 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.833352 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:25.833818 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:25.835104 (+ 1286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:25.835145 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:25.839295 (+ 4150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:25.843936 (+ 4641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:25.843950 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:25.843973 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:25.844901 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:25.845414 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:25.845430 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:25.845433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:25.845667 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:25.846289 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:25.846298 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:25.846798 (+ 500us) client_negotiation.cc:769] Sending connection context
0811 03:04:25.847159 (+ 361us) client_negotiation.cc:240] Negotiation successful
0811 03:04:25.847881 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":447,"threads_started":1}
I20260811 03:04:25.851092 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:25.820430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57613 (local address 127.27.109.62:39885)
0811 03:04:25.821002 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:25.821012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:25.822495 (+ 1483us) server_negotiation.cc:407] Connection header received
0811 03:04:25.824900 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:25.824907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:25.825012 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:25.825145 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:25.828082 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.829137 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.832524 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.832993 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.833187 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:25.839680 (+ 6493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:25.839687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:25.839691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:25.839765 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:25.843658 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:25.845100 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:25.845110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:25.845114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:25.845271 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:25.845814 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:25.845820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:25.845824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:25.846136 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:25.846314 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:25.850260 (+ 3946us) server_negotiation.cc:299] Negotiation successful
0811 03:04:25.850585 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
I20260811 03:04:25.869721 29461 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:39453
I20260811 03:04:25.869891 29620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:39453 every 8 connection(s)
I20260811 03:04:25.871234 29461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:25.873442 29621 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39885
I20260811 03:04:25.874286 29621 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:25.875298 29621 heartbeater.cc:511] Master 127.27.109.62:39885 requested a full tablet report, sending...
I20260811 03:04:25.876399 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:25.859638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44751 (local address 127.27.109.62:39885)
0811 03:04:25.859879 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:25.859890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:25.860793 (+ 903us) server_negotiation.cc:407] Connection header received
0811 03:04:25.861147 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:25.861151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:25.861226 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:25.861352 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:25.863181 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.864217 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.866740 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.867503 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.867660 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:25.870154 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:25.870160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:25.870163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:25.870201 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:25.873471 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:25.874274 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:25.874279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:25.874281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:25.874468 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:25.874978 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:25.874984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:25.874987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:25.875289 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:25.875467 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:25.875911 (+ 444us) server_negotiation.cc:299] Negotiation successful
0811 03:04:25.876096 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260811 03:04:25.877660 29378 ts_manager.cc:194] Registered new tserver with Master: bcfe39d86d56402e9327f5329c612a61 (127.27.109.1:39453)
I20260811 03:04:25.876626 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:25.859659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39885 (local address 127.27.109.1:44751)
0811 03:04:25.860637 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:25.860668 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:25.860955 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:25.861599 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:25.861604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:25.862361 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:25.863000 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:25.863015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.864470 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.864477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:25.866497 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:25.866522 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:25.867743 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:25.867754 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:25.867936 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:25.868739 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:25.868778 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:25.869838 (+ 1060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:25.873604 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:25.873608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:25.873611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:25.874152 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:25.874604 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:25.874617 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:25.874621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:25.874822 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:25.875423 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:25.875429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:25.875642 (+ 213us) client_negotiation.cc:769] Sending connection context
0811 03:04:25.875828 (+ 186us) client_negotiation.cc:240] Negotiation successful
0811 03:04:25.876206 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":569,"threads_started":1}
I20260811 03:04:25.879293 29378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:57613
I20260811 03:04:25.880426 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 29461
I20260811 03:04:25.880679 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:25.881579 28084 external_mini_cluster.cc:1428] Setting key 660fcf092424d309195eab16c4070e47
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:04:25.930899 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39885
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:26.281919 29625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:26.282248 29625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:26.282409 29625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:26.290059 29625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:26.290187 29625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:26.290349 29625 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:04:26.290442 29625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:04:26.301339 29625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:26.302814 29625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:26.304253 29625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:26.318841 29632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:26.318928 29634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:26.318948 29637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:26.320351 29625 server_base.cc:1034] running on GCE node
I20260811 03:04:26.321163 29625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:26.322378 29625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:26.323669 29625 hybrid_clock.cc:648] HybridClock initialized: now 1786417466323564 us; error 72 us; skew 500 ppm
I20260811 03:04:26.324110 29625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:26 dist-test-slave-8tdl krb5kdc[29328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:26.328136 29625 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:04:26.329882 29625 webserver.cc:460] Webserver started at http://127.27.109.2:34981/ using document root <none> and password file <none>
I20260811 03:04:26.330662 29625 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:26.330875 29625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:26.331234 29625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:26.334419 29625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "3963c9220ebc450088af555b107fe404"
format_stamp: "Formatted at 2026-08-11 03:04:26 on dist-test-slave-8tdl"
server_key: "79c0c9fdc46429f96f74168a1e871336"
server_key_iv: "4257183c05e10d30ffbfa1af8e8e01dc"
server_key_version: "encryptionkey@0"
I20260811 03:04:26.335469 29625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "3963c9220ebc450088af555b107fe404"
format_stamp: "Formatted at 2026-08-11 03:04:26 on dist-test-slave-8tdl"
server_key: "79c0c9fdc46429f96f74168a1e871336"
server_key_iv: "4257183c05e10d30ffbfa1af8e8e01dc"
server_key_version: "encryptionkey@0"
I20260811 03:04:26.341315 29625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:04:26.345599 29645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:26.347530 29625 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:04:26.347754 29625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "3963c9220ebc450088af555b107fe404"
format_stamp: "Formatted at 2026-08-11 03:04:26 on dist-test-slave-8tdl"
server_key: "79c0c9fdc46429f96f74168a1e871336"
server_key_iv: "4257183c05e10d30ffbfa1af8e8e01dc"
server_key_version: "encryptionkey@0"
I20260811 03:04:26.348026 29625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:26.366153 29625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:26.371086 29625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:26.371397 29625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:26.374701 29625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:26.374903 29625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 03:04:26.375116 29625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:26.375244 29625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:04:26 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:26.402549 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:26.375146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:41229 (local address 127.27.109.62:39885)
0811 03:04:26.375349 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:26.375358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:26.376921 (+ 1563us) server_negotiation.cc:407] Connection header received
0811 03:04:26.379264 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:26.379271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:26.379371 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:26.379498 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:26.382234 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.383457 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.386083 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.386818 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.386970 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:26.393637 (+ 6667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:26.393647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:26.393655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:26.393719 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:26.397902 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:26.399472 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:26.399482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:26.399488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:26.399685 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:26.400416 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:26.400422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:26.400426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:26.400783 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:26.400971 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:26.401877 (+ 906us) server_negotiation.cc:299] Negotiation successful
0811 03:04:26.402159 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:04:26.403775 29654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:26.375838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39885 (local address 127.27.109.2:41229)
0811 03:04:26.376733 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:26.376777 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:26.378843 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:26.379786 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:26.379803 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:26.380716 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:26.381979 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:26.382026 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.383651 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.383655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:26.385799 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:26.385830 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.387093 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.387102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:26.387389 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:26.388531 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:26.388578 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:26.393212 (+ 4634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:26.398165 (+ 4953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:26.398185 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:26.398211 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:26.399226 (+ 1015us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:26.399883 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:26.399896 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:26.399901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:26.400251 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:26.400959 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:26.400969 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:26.401399 (+ 430us) client_negotiation.cc:769] Sending connection context
0811 03:04:26.401719 (+ 320us) client_negotiation.cc:240] Negotiation successful
0811 03:04:26.402360 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":353,"threads_started":1}
I20260811 03:04:26.434303 29625 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:33959
I20260811 03:04:26.434325 29777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:33959 every 8 connection(s)
I20260811 03:04:26.435880 29625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:04:26.438301 29778 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39885
I20260811 03:04:26.438567 29778 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:26.439332 29778 heartbeater.cc:511] Master 127.27.109.62:39885 requested a full tablet report, sending...
I20260811 03:04:26.439721 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:26.422029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:38569 (local address 127.27.109.62:39885)
0811 03:04:26.422524 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:26.422529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:26.423146 (+ 617us) server_negotiation.cc:407] Connection header received
0811 03:04:26.423393 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:26.423399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:26.423476 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:26.423581 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:26.425268 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.426213 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.429014 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.429635 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.429801 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:26.432797 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:26.432804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:26.432811 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:26.432856 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:26.436471 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:26.437254 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:26.437259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:26.437263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:26.437462 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:26.437974 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:26.437978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:26.437981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:26.438485 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:26.438667 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:26.439136 (+ 469us) server_negotiation.cc:299] Negotiation successful
0811 03:04:26.439373 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
I20260811 03:04:26.439869 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:26.422156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39885 (local address 127.27.109.2:38569)
0811 03:04:26.422996 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:26.423024 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:26.423224 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:26.423655 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:26.423662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:26.424540 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:26.425038 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:26.425056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.426368 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.426377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:26.428679 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:26.428700 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.430106 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.430119 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:26.430380 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:26.431255 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:26.431275 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:26.432449 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:26.436634 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:26.436640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:26.436644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:26.437105 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:26.437618 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:26.437625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:26.437632 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:26.437798 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:26.438615 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:26.438620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:26.438851 (+ 231us) client_negotiation.cc:769] Sending connection context
0811 03:04:26.439012 (+ 161us) client_negotiation.cc:240] Negotiation successful
0811 03:04:26.439393 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":384,"threads_started":1}
I20260811 03:04:26.441073 29378 ts_manager.cc:194] Registered new tserver with Master: 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959)
I20260811 03:04:26.442000 29378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:41229
I20260811 03:04:26.444886 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 29625
I20260811 03:04:26.445120 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:04:26.445639 28084 external_mini_cluster.cc:1428] Setting key 53eae3d7ee4e03d3455e3ca034ad391c
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:04:26.496663 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39885
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:26.869925 29782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:26.870389 29782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:26.870555 29782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:26.880321 29782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:26.880452 29782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:26.880542 29782 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:04:26.880625 29782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:04:26.882921 29621 heartbeater.cc:503] Master 127.27.109.62:39885 was elected leader, sending a full tablet report...
I20260811 03:04:26.891577 29782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:39885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:26.892999 29782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:26.894459 29782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:26.909881 29787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:26.909962 29792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:26.909922 29788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:26.911062 29782 server_base.cc:1034] running on GCE node
I20260811 03:04:26.911737 29782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:26.912766 29782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:26.914052 29782 hybrid_clock.cc:648] HybridClock initialized: now 1786417466913988 us; error 43 us; skew 500 ppm
I20260811 03:04:26.914494 29782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:26 dist-test-slave-8tdl krb5kdc[29328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:26.918045 29782 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:04:26.919863 29782 webserver.cc:460] Webserver started at http://127.27.109.3:41875/ using document root <none> and password file <none>
I20260811 03:04:26.920653 29782 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:26.920799 29782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:26.921159 29782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:26.924445 29782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "c38fae8d46384e6d8a650994c1b6adcb"
format_stamp: "Formatted at 2026-08-11 03:04:26 on dist-test-slave-8tdl"
server_key: "24ef5b6db6fbe60ce2b8795a7b355e22"
server_key_iv: "c889a073209ae9a0b562d3835cb6f4f6"
server_key_version: "encryptionkey@0"
I20260811 03:04:26.925346 29782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "c38fae8d46384e6d8a650994c1b6adcb"
format_stamp: "Formatted at 2026-08-11 03:04:26 on dist-test-slave-8tdl"
server_key: "24ef5b6db6fbe60ce2b8795a7b355e22"
server_key_iv: "c889a073209ae9a0b562d3835cb6f4f6"
server_key_version: "encryptionkey@0"
I20260811 03:04:26.931447 29782 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260811 03:04:26.935463 29800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:26.937240 29782 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 03:04:26.937467 29782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "c38fae8d46384e6d8a650994c1b6adcb"
format_stamp: "Formatted at 2026-08-11 03:04:26 on dist-test-slave-8tdl"
server_key: "24ef5b6db6fbe60ce2b8795a7b355e22"
server_key_iv: "c889a073209ae9a0b562d3835cb6f4f6"
server_key_version: "encryptionkey@0"
I20260811 03:04:26.937726 29782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:26.953428 29782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:26.958604 29782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:26.958964 29782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:26.962224 29782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:26.962344 29782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:26.962523 29782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:26.962667 29782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 11 03:04:26 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:26.992054 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:26.962804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:45295 (local address 127.27.109.62:39885)
0811 03:04:26.965553 (+ 2749us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:26.965561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:26.965604 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:04:26.967406 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:26.967413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:26.967517 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:26.967693 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:26.970591 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.971726 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.975538 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.976118 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.976292 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:26.982759 (+ 6467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:26.982770 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:26.982777 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:26.982839 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:26.987448 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:26.988741 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:26.988748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:26.988752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:26.988910 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:26.989604 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:26.989611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:26.989616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:26.989938 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:26.990144 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:26.991383 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0811 03:04:26.991627 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2607,"thread_start_us":2492,"threads_started":1}
I20260811 03:04:26.993184 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:26.963588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39885 (local address 127.27.109.3:45295)
0811 03:04:26.964750 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:26.964830 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:26.967053 (+ 2223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:26.968131 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:26.968148 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:26.969087 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:26.970197 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:26.970378 (+ 181us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.971975 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.971988 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:26.975203 (+ 3215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:26.975232 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:26.976446 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:26.976459 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:26.976936 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:26.978034 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:26.978074 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:26.982354 (+ 4280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:26.987696 (+ 5342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:26.987714 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:26.987741 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:26.988577 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:26.989041 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:26.989047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:26.989051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:26.989401 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:26.990247 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:26.990258 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:26.990849 (+ 591us) client_negotiation.cc:769] Sending connection context
0811 03:04:26.991255 (+ 406us) client_negotiation.cc:240] Negotiation successful
0811 03:04:26.991882 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":465,"threads_started":1}
I20260811 03:04:27.013676 29782 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:45809
I20260811 03:04:27.013723 29937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:45809 every 8 connection(s)
I20260811 03:04:27.015232 29782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:04:27.017769 29938 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39885
I20260811 03:04:27.018041 29938 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:27.018998 29938 heartbeater.cc:511] Master 127.27.109.62:39885 requested a full tablet report, sending...
I20260811 03:04:27.019366 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.001390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:50159 (local address 127.27.109.62:39885)
0811 03:04:27.001627 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.001637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.002538 (+ 901us) server_negotiation.cc:407] Connection header received
0811 03:04:27.002846 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.002855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.003064 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.003201 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:27.004979 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.006182 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.008377 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.009110 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.009282 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.012157 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:27.012167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:27.012175 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:27.012233 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:27.015588 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:27.016841 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:27.016847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:27.016850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:27.017012 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:27.017458 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:27.017462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:27.017464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:27.017843 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:27.018078 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.018638 (+ 560us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.019003 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 03:04:27.019454 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.001424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39885 (local address 127.27.109.3:50159)
0811 03:04:27.002352 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.002386 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.002631 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.003249 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.003256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:27.004179 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:27.004794 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.004809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.006360 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.006368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.008130 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.008162 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.009326 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.009333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.009506 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.010321 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:27.010358 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:27.011857 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:27.015768 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:27.015778 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:27.015786 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:27.016646 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:27.017151 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:27.017156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:27.017160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:27.017331 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:27.018058 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:27.018066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:27.018300 (+ 234us) client_negotiation.cc:769] Sending connection context
0811 03:04:27.018471 (+ 171us) client_negotiation.cc:240] Negotiation successful
0811 03:04:27.018894 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":465,"threads_started":1}
I20260811 03:04:27.020797 29376 ts_manager.cc:194] Registered new tserver with Master: c38fae8d46384e6d8a650994c1b6adcb (127.27.109.3:45809)
I20260811 03:04:27.021576 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 29782
I20260811 03:04:27.021788 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:04:27.022192 29376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:45295
I20260811 03:04:27.022290 28084 external_mini_cluster.cc:1428] Setting key 0ec571479cd1cc26c8925370511f7408
I20260811 03:04:27.025804 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 03:04:27 dist-test-slave-8tdl krb5kdc[29328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417467, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 03:04:27 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417467, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:27.083717 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.067761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.27.109.62:39885)
0811 03:04:27.068021 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.068029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.068242 (+ 213us) server_negotiation.cc:407] Connection header received
0811 03:04:27.068482 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.068487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.068556 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.068652 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:27.070202 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.071041 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.072331 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.072808 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.072927 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.077678 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:27.077684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:27.077688 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:27.077731 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:27.080755 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:27.081487 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:27.081492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:27.081494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:27.081645 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:27.082051 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:27.082055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:27.082057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:27.082366 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:27.082611 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.083076 (+ 465us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.083268 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260811 03:04:27.089205 29376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:04:27.097684 29376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:27.125552 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.111534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39453 (local address 127.0.0.1:32848)
0811 03:04:27.114835 (+ 3301us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.114921 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.115343 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.116600 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.116612 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:27.116659 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:27.117241 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.117255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.119325 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.119334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.122066 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.122083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.124461 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.124467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.124693 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.124752 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 03:04:27.124916 (+ 164us) client_negotiation.cc:240] Negotiation successful
0811 03:04:27.125127 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2904,"thread_start_us":2410,"threads_started":1}
I20260811 03:04:27.125583 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.111855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.27.109.1:39453)
0811 03:04:27.112807 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.112818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.115185 (+ 2367us) server_negotiation.cc:407] Connection header received
0811 03:04:27.115720 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.115734 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.116096 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.116337 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:27.117447 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.119140 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.122264 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.123930 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.124083 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.124310 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.125007 (+ 697us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.125231 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":596,"threads_started":1}
I20260811 03:04:27.126698 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.112836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:45809 (local address 127.0.0.1:47258)
0811 03:04:27.117119 (+ 4283us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.117140 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.117431 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.118131 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.118137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:27.118166 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:27.118746 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.118761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.120981 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.120985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.123127 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.123147 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.125711 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.125718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.125925 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.125964 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 03:04:27.126085 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 03:04:27.126236 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4078,"mutex_wait_us":28,"thread_start_us":4100,"threads_started":1}
I20260811 03:04:27.126739 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.113231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.27.109.2:33959)
0811 03:04:27.113904 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.113916 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.117468 (+ 3552us) server_negotiation.cc:407] Connection header received
0811 03:04:27.117623 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.117631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.117908 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.118118 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:27.119173 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.120771 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.123416 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.124813 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.124957 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.125152 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.126212 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.126422 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":351,"threads_started":1}
I20260811 03:04:27.126752 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.112365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.27.109.3:45809)
0811 03:04:27.113298 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.113309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.117276 (+ 3967us) server_negotiation.cc:407] Connection header received
0811 03:04:27.117700 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.117707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.117912 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.118118 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:27.119004 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.120770 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.123321 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.125134 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.125276 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.125529 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.126151 (+ 622us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.126343 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":571,"threads_started":1}
I20260811 03:04:27.127177 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.113346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33959 (local address 127.0.0.1:40516)
0811 03:04:27.117302 (+ 3956us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.117322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.117458 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.118386 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.118391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:27.118417 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:27.118911 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.118930 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.120936 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.120943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.123203 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.123244 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.125310 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.125317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.125492 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.125539 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 03:04:27.125628 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 03:04:27.125797 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3818,"mutex_wait_us":23,"thread_start_us":3330,"threads_started":1}
I20260811 03:04:27.130425 29540 tablet_service.cc:1467] Processing CreateTablet for tablet c034213b72ba4c13866807c44aec64ac (DEFAULT_TABLE table=test-table [id=b11a26bece31404185c00f9a4aa50fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:27.130951 29701 tablet_service.cc:1467] Processing CreateTablet for tablet c034213b72ba4c13866807c44aec64ac (DEFAULT_TABLE table=test-table [id=b11a26bece31404185c00f9a4aa50fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:27.132159 29540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c034213b72ba4c13866807c44aec64ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:27.132297 29701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c034213b72ba4c13866807c44aec64ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:27.133100 29860 tablet_service.cc:1467] Processing CreateTablet for tablet c034213b72ba4c13866807c44aec64ac (DEFAULT_TABLE table=test-table [id=b11a26bece31404185c00f9a4aa50fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:27.134367 29860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c034213b72ba4c13866807c44aec64ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:27.147846 29959 tablet_bootstrap.cc:492] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Bootstrap starting.
I20260811 03:04:27.150413 29961 tablet_bootstrap.cc:492] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Bootstrap starting.
I20260811 03:04:27.151463 29962 tablet_bootstrap.cc:492] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Bootstrap starting.
I20260811 03:04:27.154117 29959 tablet_bootstrap.cc:654] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:27.155805 29959 log.cc:824] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:27.156819 29961 tablet_bootstrap.cc:654] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:27.157286 29962 tablet_bootstrap.cc:654] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:27.158010 29961 log.cc:824] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:27.158748 29962 log.cc:824] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:27.159718 29959 tablet_bootstrap.cc:492] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: No bootstrap required, opened a new log
I20260811 03:04:27.160032 29959 ts_tablet_manager.cc:1397] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260811 03:04:27.161011 29961 tablet_bootstrap.cc:492] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: No bootstrap required, opened a new log
I20260811 03:04:27.161329 29961 ts_tablet_manager.cc:1397] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260811 03:04:27.162073 29962 tablet_bootstrap.cc:492] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: No bootstrap required, opened a new log
I20260811 03:04:27.162361 29962 ts_tablet_manager.cc:1397] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260811 03:04:27.170100 29961 raft_consensus.cc:348] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.170562 29961 raft_consensus.cc:374] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:27.170760 29961 raft_consensus.cc:729] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963c9220ebc450088af555b107fe404, State: Initialized, Role: FOLLOWER
I20260811 03:04:27.171464 29961 consensus_queue.cc:260] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.172619 29778 heartbeater.cc:503] Master 127.27.109.62:39885 was elected leader, sending a full tablet report...
I20260811 03:04:27.173261 29959 raft_consensus.cc:348] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.173851 29959 raft_consensus.cc:374] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:27.174016 29959 raft_consensus.cc:729] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c38fae8d46384e6d8a650994c1b6adcb, State: Initialized, Role: FOLLOWER
I20260811 03:04:27.174566 29961 ts_tablet_manager.cc:1428] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260811 03:04:27.174502 29962 raft_consensus.cc:348] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.175140 29962 raft_consensus.cc:374] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:27.175302 29962 raft_consensus.cc:729] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfe39d86d56402e9327f5329c612a61, State: Initialized, Role: FOLLOWER
I20260811 03:04:27.175325 29959 consensus_queue.cc:260] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.176611 29938 heartbeater.cc:503] Master 127.27.109.62:39885 was elected leader, sending a full tablet report...
I20260811 03:04:27.177814 29959 ts_tablet_manager.cc:1428] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260811 03:04:27.176173 29962 consensus_queue.cc:260] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.179960 29962 ts_tablet_manager.cc:1428] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
W20260811 03:04:27.188817 29779 tablet.cc:2369] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:04:27.267217 29939 tablet.cc:2369] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:04:27.374243 29622 tablet.cc:2369] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:04:27.434557 29970 raft_consensus.cc:482] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:27.435055 29970 raft_consensus.cc:504] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.437070 29970 leader_election.cc:288] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c38fae8d46384e6d8a650994c1b6adcb (127.27.109.3:45809), 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959)
I20260811 03:04:27.445817 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.438384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:42597 (local address 127.27.109.3:45809)
0811 03:04:27.438629 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.438637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.439588 (+ 951us) server_negotiation.cc:407] Connection header received
0811 03:04:27.439907 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.439911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.440020 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.440119 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:27.440815 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.441690 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.443879 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.444794 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.444975 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.445060 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.445387 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.445541 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 03:04:27.445902 29973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.437304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:45809 (local address 127.27.109.1:42597)
0811 03:04:27.439443 (+ 2139us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.439471 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.439746 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.440164 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.440169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:27.440195 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:27.440637 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.440649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.441825 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.441833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.443718 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.443730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.445015 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.445021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.445182 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.445216 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:04:27.445314 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 03:04:27.445472 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1927,"mutex_wait_us":28,"thread_start_us":2039,"threads_started":1}
I20260811 03:04:27.447206 29881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c034213b72ba4c13866807c44aec64ac" candidate_uuid: "bcfe39d86d56402e9327f5329c612a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c38fae8d46384e6d8a650994c1b6adcb" is_pre_election: true
I20260811 03:04:27.447427 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.437743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44493 (local address 127.27.109.2:33959)
0811 03:04:27.438007 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.438016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.439772 (+ 1756us) server_negotiation.cc:407] Connection header received
0811 03:04:27.440004 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.440010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.440105 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.440189 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:27.440980 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.442140 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.444724 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.446015 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.446202 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.446279 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.447013 (+ 734us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.447169 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:04:27.447463 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.437566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33959 (local address 127.27.109.1:44493)
0811 03:04:27.439640 (+ 2074us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.439662 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.439800 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.440365 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.440368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:27.440388 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:27.440810 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.440821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.442346 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.442351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.444544 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.444559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.446598 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.446604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.446825 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.446861 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 03:04:27.446979 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 03:04:27.447122 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1938,"thread_start_us":1379,"threads_started":1}
I20260811 03:04:27.447857 29881 raft_consensus.cc:2454] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcfe39d86d56402e9327f5329c612a61 in term 0.
I20260811 03:04:27.448603 29480 leader_election.cc:302] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcfe39d86d56402e9327f5329c612a61, c38fae8d46384e6d8a650994c1b6adcb; no voters:
I20260811 03:04:27.448789 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c034213b72ba4c13866807c44aec64ac" candidate_uuid: "bcfe39d86d56402e9327f5329c612a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963c9220ebc450088af555b107fe404" is_pre_election: true
I20260811 03:04:27.449098 29970 raft_consensus.cc:2781] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:27.449255 29970 raft_consensus.cc:482] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:27.449345 29970 raft_consensus.cc:3037] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:27.449424 29727 raft_consensus.cc:2454] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcfe39d86d56402e9327f5329c612a61 in term 0.
I20260811 03:04:27.452416 29970 raft_consensus.cc:504] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.453171 29970 leader_election.cc:288] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [CANDIDATE]: Term 1 election: Requested vote from peers c38fae8d46384e6d8a650994c1b6adcb (127.27.109.3:45809), 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959)
I20260811 03:04:27.453719 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c034213b72ba4c13866807c44aec64ac" candidate_uuid: "bcfe39d86d56402e9327f5329c612a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963c9220ebc450088af555b107fe404"
I20260811 03:04:27.453719 29881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c034213b72ba4c13866807c44aec64ac" candidate_uuid: "bcfe39d86d56402e9327f5329c612a61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c38fae8d46384e6d8a650994c1b6adcb"
I20260811 03:04:27.453948 29727 raft_consensus.cc:3037] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:27.453948 29881 raft_consensus.cc:3037] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:27.457300 29727 raft_consensus.cc:2454] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcfe39d86d56402e9327f5329c612a61 in term 1.
I20260811 03:04:27.457300 29881 raft_consensus.cc:2454] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcfe39d86d56402e9327f5329c612a61 in term 1.
I20260811 03:04:27.457741 29480 leader_election.cc:302] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3963c9220ebc450088af555b107fe404, bcfe39d86d56402e9327f5329c612a61, c38fae8d46384e6d8a650994c1b6adcb; no voters:
I20260811 03:04:27.458050 29970 raft_consensus.cc:2781] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:27.458509 29970 raft_consensus.cc:686] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 1 LEADER]: Becoming Leader. State: Replica: bcfe39d86d56402e9327f5329c612a61, State: Running, Role: LEADER
I20260811 03:04:27.459044 29970 consensus_queue.cc:237] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } }
I20260811 03:04:27.464188 29378 catalog_manager.cc:5495] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcfe39d86d56402e9327f5329c612a61 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "bcfe39d86d56402e9327f5329c612a61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } health_report { overall_health: UNKNOWN } } }
I20260811 03:04:27.502264 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.496226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.27.109.1:39453)
0811 03:04:27.496456 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.496462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.496588 (+ 126us) server_negotiation.cc:407] Connection header received
0811 03:04:27.497044 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.497049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.497178 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.497332 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:27.497712 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.498493 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.499886 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.500303 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.500398 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.500913 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:27.501601 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:27.501728 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.501885 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.502020 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 03:04:27.512106 29727 raft_consensus.cc:1261] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Refusing update from remote peer bcfe39d86d56402e9327f5329c612a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:27.512106 29881 raft_consensus.cc:1261] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Refusing update from remote peer bcfe39d86d56402e9327f5329c612a61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:27.513314 29970 consensus_queue.cc:1035] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:27.513690 29974 consensus_queue.cc:1035] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:27.529578 29977 mvcc.cc:205] Tried to move back new op lower bound from 7317165946918162432 to 7317165946716262400. Current Snapshot: MvccSnapshot[applied={T|T < 7317165946918162432}]
I20260811 03:04:27.551389 29378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:27.551764 29378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:27.556675 29378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd: Sending DeleteTablet for 3 replicas of tablet c034213b72ba4c13866807c44aec64ac
I20260811 03:04:27.558283 29860 tablet_service.cc:1514] Processing DeleteTablet for tablet c034213b72ba4c13866807c44aec64ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:27 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:47258
I20260811 03:04:27.558276 29540 tablet_service.cc:1514] Processing DeleteTablet for tablet c034213b72ba4c13866807c44aec64ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:27 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:32848
I20260811 03:04:27.559391 29701 tablet_service.cc:1514] Processing DeleteTablet for tablet c034213b72ba4c13866807c44aec64ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:27 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40516
I20260811 03:04:27.564503 29378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:04:27.565634 29378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:27.575264 29988 tablet_replica.cc:330] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: stopping tablet replica
I20260811 03:04:27.576726 29988 raft_consensus.cc:2229] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:04:27.578001 29989 tablet_replica.cc:330] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: stopping tablet replica
I20260811 03:04:27.578728 29989 raft_consensus.cc:2229] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:27.578855 29988 raft_consensus.cc:2258] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:27.579537 29989 raft_consensus.cc:2258] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:27.581168 29539 tablet_service.cc:1467] Processing CreateTablet for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 (DEFAULT_TABLE table=test-table [id=996b010c712c4121b48893925c71ec00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:27.581389 29858 tablet_service.cc:1467] Processing CreateTablet for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 (DEFAULT_TABLE table=test-table [id=996b010c712c4121b48893925c71ec00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:27.581857 29539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:27.581884 29858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:27.587138 29701 tablet_service.cc:1467] Processing CreateTablet for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 (DEFAULT_TABLE table=test-table [id=996b010c712c4121b48893925c71ec00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:27.587786 29701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:27.590245 29962 tablet_bootstrap.cc:492] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: Bootstrap starting.
I20260811 03:04:27.592654 29962 tablet_bootstrap.cc:654] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:27.593495 29991 tablet_replica.cc:330] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: stopping tablet replica
I20260811 03:04:27.594005 29991 raft_consensus.cc:2229] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:27.596043 29959 tablet_bootstrap.cc:492] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: Bootstrap starting.
I20260811 03:04:27.596387 29991 raft_consensus.cc:2258] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:27.598127 29959 tablet_bootstrap.cc:654] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:27.602443 29991 ts_tablet_manager.cc:1905] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:27.605230 29989 ts_tablet_manager.cc:1905] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:27.616336 29991 ts_tablet_manager.cc:1918] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:04:27.616580 29991 log.cc:1196] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/c034213b72ba4c13866807c44aec64ac
I20260811 03:04:27.617187 29991 ts_tablet_manager.cc:1939] T c034213b72ba4c13866807c44aec64ac P c38fae8d46384e6d8a650994c1b6adcb: Deleting consensus metadata
I20260811 03:04:27.619395 29360 catalog_manager.cc:4841] TS c38fae8d46384e6d8a650994c1b6adcb (127.27.109.3:45809): tablet c034213b72ba4c13866807c44aec64ac (table test-table [id=b11a26bece31404185c00f9a4aa50fc5]) successfully deleted
I20260811 03:04:27.621661 29988 ts_tablet_manager.cc:1905] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:27.626405 29959 tablet_bootstrap.cc:492] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: No bootstrap required, opened a new log
I20260811 03:04:27.626632 29959 ts_tablet_manager.cc:1397] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20260811 03:04:27.626653 29989 ts_tablet_manager.cc:1918] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:04:27.626900 29989 log.cc:1196] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/c034213b72ba4c13866807c44aec64ac
I20260811 03:04:27.627461 29961 tablet_bootstrap.cc:492] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: Bootstrap starting.
I20260811 03:04:27.627765 29959 raft_consensus.cc:348] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.628041 29959 raft_consensus.cc:374] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:27.628159 29959 raft_consensus.cc:729] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c38fae8d46384e6d8a650994c1b6adcb, State: Initialized, Role: FOLLOWER
I20260811 03:04:27.628386 29962 tablet_bootstrap.cc:492] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: No bootstrap required, opened a new log
I20260811 03:04:27.628614 29962 ts_tablet_manager.cc:1397] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.023s
I20260811 03:04:27.628799 29959 consensus_queue.cc:260] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.629499 29961 tablet_bootstrap.cc:654] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:27.629536 29962 raft_consensus.cc:348] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.629783 29962 raft_consensus.cc:374] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:27.629911 29962 raft_consensus.cc:729] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfe39d86d56402e9327f5329c612a61, State: Initialized, Role: FOLLOWER
I20260811 03:04:27.630381 29989 ts_tablet_manager.cc:1939] T c034213b72ba4c13866807c44aec64ac P 3963c9220ebc450088af555b107fe404: Deleting consensus metadata
I20260811 03:04:27.631737 29959 ts_tablet_manager.cc:1428] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260811 03:04:27.632418 29362 catalog_manager.cc:4841] TS 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959): tablet c034213b72ba4c13866807c44aec64ac (table test-table [id=b11a26bece31404185c00f9a4aa50fc5]) successfully deleted
I20260811 03:04:27.632817 29962 consensus_queue.cc:260] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.634253 29962 ts_tablet_manager.cc:1428] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260811 03:04:27.636149 29961 tablet_bootstrap.cc:492] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: No bootstrap required, opened a new log
I20260811 03:04:27.636328 29961 ts_tablet_manager.cc:1397] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260811 03:04:27.637209 29961 raft_consensus.cc:348] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.637468 29961 raft_consensus.cc:374] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:27.637578 29961 raft_consensus.cc:729] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963c9220ebc450088af555b107fe404, State: Initialized, Role: FOLLOWER
I20260811 03:04:27.637930 29961 consensus_queue.cc:260] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.638549 29961 ts_tablet_manager.cc:1428] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260811 03:04:27.639108 29988 ts_tablet_manager.cc:1918] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:04:27.639354 29988 log.cc:1196] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/c034213b72ba4c13866807c44aec64ac
I20260811 03:04:27.640098 29988 ts_tablet_manager.cc:1939] T c034213b72ba4c13866807c44aec64ac P bcfe39d86d56402e9327f5329c612a61: Deleting consensus metadata
I20260811 03:04:27.642249 29359 catalog_manager.cc:4841] TS bcfe39d86d56402e9327f5329c612a61 (127.27.109.1:39453): tablet c034213b72ba4c13866807c44aec64ac (table test-table [id=b11a26bece31404185c00f9a4aa50fc5]) successfully deleted
I20260811 03:04:27.884033 29968 raft_consensus.cc:482] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:27.884320 29968 raft_consensus.cc:504] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.886338 29968 leader_election.cc:288] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcfe39d86d56402e9327f5329c612a61 (127.27.109.1:39453), 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959)
I20260811 03:04:27.899516 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.887058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:41447 (local address 127.27.109.2:33959)
0811 03:04:27.887432 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.887439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.892075 (+ 4636us) server_negotiation.cc:407] Connection header received
0811 03:04:27.892343 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.892347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.892456 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.892554 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:27.893407 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.894459 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.896833 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.898192 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.898367 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.898449 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.899005 (+ 556us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.899168 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260811 03:04:27.899884 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.887426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:38015 (local address 127.27.109.1:39453)
0811 03:04:27.887648 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.887655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.891776 (+ 4121us) server_negotiation.cc:407] Connection header received
0811 03:04:27.892150 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.892164 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.892280 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.892417 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:27.893054 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.894199 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.897226 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.898648 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.898827 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.898898 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:27.899403 (+ 505us) server_negotiation.cc:299] Negotiation successful
0811 03:04:27.899554 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 03:04:27.900439 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.887473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:33959 (local address 127.27.109.3:41447)
0811 03:04:27.891954 (+ 4481us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.891973 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.892177 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.892651 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.892657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:27.892689 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:27.893186 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.893203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.894605 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.894614 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.896628 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.896647 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.898576 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.898583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.898803 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.898842 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 03:04:27.899802 (+ 960us) client_negotiation.cc:240] Negotiation successful
0811 03:04:27.899968 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4358,"mutex_wait_us":3,"thread_start_us":3760,"threads_started":1}
I20260811 03:04:27.900583 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.886657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39453 (local address 127.27.109.3:38015)
0811 03:04:27.891603 (+ 4946us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.891633 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.891921 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.892433 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.892439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:27.892465 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:27.892880 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.892892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.894369 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.894376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.897036 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:27.897056 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:27.899050 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:27.899057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:27.899214 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:27.899246 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 03:04:27.900083 (+ 837us) client_negotiation.cc:240] Negotiation successful
0811 03:04:27.900216 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4734,"mutex_wait_us":55,"thread_start_us":4933,"threads_started":1}
I20260811 03:04:27.901333 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc393c8b2f4dc0a2f5b34ecb4669d6" candidate_uuid: "c38fae8d46384e6d8a650994c1b6adcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963c9220ebc450088af555b107fe404" is_pre_election: true
I20260811 03:04:27.901628 29727 raft_consensus.cc:2454] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38fae8d46384e6d8a650994c1b6adcb in term 0.
I20260811 03:04:27.901793 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc393c8b2f4dc0a2f5b34ecb4669d6" candidate_uuid: "c38fae8d46384e6d8a650994c1b6adcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfe39d86d56402e9327f5329c612a61" is_pre_election: true
I20260811 03:04:27.902184 29803 leader_election.cc:302] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3963c9220ebc450088af555b107fe404, c38fae8d46384e6d8a650994c1b6adcb; no voters:
I20260811 03:04:27.902320 29561 raft_consensus.cc:2454] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38fae8d46384e6d8a650994c1b6adcb in term 0.
I20260811 03:04:27.902695 29968 raft_consensus.cc:2781] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:27.902834 29968 raft_consensus.cc:482] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:27.902956 29968 raft_consensus.cc:3037] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:27.905292 29968 raft_consensus.cc:504] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.906031 29968 leader_election.cc:288] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [CANDIDATE]: Term 1 election: Requested vote from peers bcfe39d86d56402e9327f5329c612a61 (127.27.109.1:39453), 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959)
I20260811 03:04:27.906625 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc393c8b2f4dc0a2f5b34ecb4669d6" candidate_uuid: "c38fae8d46384e6d8a650994c1b6adcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfe39d86d56402e9327f5329c612a61"
I20260811 03:04:27.906651 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc393c8b2f4dc0a2f5b34ecb4669d6" candidate_uuid: "c38fae8d46384e6d8a650994c1b6adcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963c9220ebc450088af555b107fe404"
I20260811 03:04:27.906949 29561 raft_consensus.cc:3037] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:27.906944 29727 raft_consensus.cc:3037] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:27.909469 29561 raft_consensus.cc:2454] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38fae8d46384e6d8a650994c1b6adcb in term 1.
I20260811 03:04:27.909986 29801 leader_election.cc:302] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcfe39d86d56402e9327f5329c612a61, c38fae8d46384e6d8a650994c1b6adcb; no voters:
I20260811 03:04:27.910210 29727 raft_consensus.cc:2454] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38fae8d46384e6d8a650994c1b6adcb in term 1.
I20260811 03:04:27.910324 29968 raft_consensus.cc:2781] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:27.910898 29968 raft_consensus.cc:686] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 1 LEADER]: Becoming Leader. State: Replica: c38fae8d46384e6d8a650994c1b6adcb, State: Running, Role: LEADER
I20260811 03:04:27.911577 29968 consensus_queue.cc:237] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:27.915256 29376 catalog_manager.cc:5495] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb reported cstate change: term changed from 0 to 1, leader changed from <none> to c38fae8d46384e6d8a650994c1b6adcb (127.27.109.3). New cstate: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } health_report { overall_health: HEALTHY } } }
Aug 11 03:04:28 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:28.019568 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.997956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39885 (local address 127.0.0.1:49336)
0811 03:04:27.998728 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:27.998758 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:27.998917 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:27.999802 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:27.999806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:28.000591 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:28.000821 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:28.000832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.002368 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.002381 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:28.003657 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:28.003673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.004798 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.004804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:28.004992 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.010637 (+ 5645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:28.010664 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:28.013414 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:28.016872 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:28.016890 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:28.016898 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:28.017381 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:28.017784 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:28.017789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:28.017792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:28.017941 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:28.018502 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:28.018510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:28.018741 (+ 231us) client_negotiation.cc:769] Sending connection context
0811 03:04:28.018953 (+ 212us) client_negotiation.cc:240] Negotiation successful
0811 03:04:28.019268 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":382,"threads_started":1}
I20260811 03:04:28.020520 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:27.997956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.27.109.62:39885)
0811 03:04:27.999446 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:27.999452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:27.999475 (+ 23us) server_negotiation.cc:407] Connection header received
0811 03:04:27.999532 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:27.999538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:27.999619 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:27.999733 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:28.001011 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.002139 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.003827 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.004562 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.004709 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.013677 (+ 8968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:28.013683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:28.013686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:28.013724 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:28.016707 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.017523 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.017533 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.017535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.017663 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.018059 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.018069 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.018071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.018349 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:28.018535 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.019955 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.020202 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362,"thread_start_us":1207,"threads_started":1}
I20260811 03:04:28.025794 29376 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49336:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 03:04:28.027024 29376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:28.036511 29701 tablet_service.cc:1467] Processing CreateTablet for tablet 61c5605491b04055b8b52f8707294304 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ac3c16a7b9d4ce98b7c36e1cce9d584]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:04:28.036900 29858 tablet_service.cc:1467] Processing CreateTablet for tablet 61c5605491b04055b8b52f8707294304 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ac3c16a7b9d4ce98b7c36e1cce9d584]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:04:28.037237 29701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c5605491b04055b8b52f8707294304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:28.037503 29858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c5605491b04055b8b52f8707294304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:28.039947 29540 tablet_service.cc:1467] Processing CreateTablet for tablet 61c5605491b04055b8b52f8707294304 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3ac3c16a7b9d4ce98b7c36e1cce9d584]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:04:28.040516 29540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c5605491b04055b8b52f8707294304. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:28.046023 29959 tablet_bootstrap.cc:492] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: Bootstrap starting.
I20260811 03:04:28.048134 29959 tablet_bootstrap.cc:654] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:28.049039 29962 tablet_bootstrap.cc:492] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: Bootstrap starting.
I20260811 03:04:28.051677 29962 tablet_bootstrap.cc:654] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:28.052472 29961 tablet_bootstrap.cc:492] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: Bootstrap starting.
I20260811 03:04:28.054342 29961 tablet_bootstrap.cc:654] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:28.054833 29959 tablet_bootstrap.cc:492] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: No bootstrap required, opened a new log
I20260811 03:04:28.055058 29959 ts_tablet_manager.cc:1397] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260811 03:04:28.055935 29959 raft_consensus.cc:348] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.056226 29959 raft_consensus.cc:374] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:28.056257 29962 tablet_bootstrap.cc:492] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: No bootstrap required, opened a new log
I20260811 03:04:28.056332 29959 raft_consensus.cc:729] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c38fae8d46384e6d8a650994c1b6adcb, State: Initialized, Role: FOLLOWER
I20260811 03:04:28.056428 29962 ts_tablet_manager.cc:1397] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 03:04:28.056607 29959 consensus_queue.cc:260] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.057199 29959 ts_tablet_manager.cc:1428] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 03:04:28.057199 29962 raft_consensus.cc:348] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.057250 29968 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.057447 29962 raft_consensus.cc:374] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:28.057546 29962 raft_consensus.cc:729] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcfe39d86d56402e9327f5329c612a61, State: Initialized, Role: FOLLOWER
I20260811 03:04:28.057617 29968 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:28.057827 29962 consensus_queue.cc:260] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.058137 29961 tablet_bootstrap.cc:492] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: No bootstrap required, opened a new log
I20260811 03:04:28.058305 29961 ts_tablet_manager.cc:1397] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260811 03:04:28.058538 29962 ts_tablet_manager.cc:1428] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260811 03:04:28.058352 29985 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.058717 29985 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:28.059093 29961 raft_consensus.cc:348] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.059362 29961 raft_consensus.cc:374] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:28.059456 29961 raft_consensus.cc:729] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3963c9220ebc450088af555b107fe404, State: Initialized, Role: FOLLOWER
I20260811 03:04:28.059737 29961 consensus_queue.cc:260] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.060340 29961 ts_tablet_manager.cc:1428] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 03:04:28.060738 29967 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.061143 29967 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:28.181316 29968 raft_consensus.cc:482] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:28.181619 29968 raft_consensus.cc:504] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.182349 29968 leader_election.cc:288] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcfe39d86d56402e9327f5329c612a61 (127.27.109.1:39453), 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959)
I20260811 03:04:28.183116 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c5605491b04055b8b52f8707294304" candidate_uuid: "c38fae8d46384e6d8a650994c1b6adcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfe39d86d56402e9327f5329c612a61" is_pre_election: true
I20260811 03:04:28.183162 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c5605491b04055b8b52f8707294304" candidate_uuid: "c38fae8d46384e6d8a650994c1b6adcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963c9220ebc450088af555b107fe404" is_pre_election: true
I20260811 03:04:28.183452 29561 raft_consensus.cc:2454] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38fae8d46384e6d8a650994c1b6adcb in term 0.
I20260811 03:04:28.183483 29727 raft_consensus.cc:2454] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38fae8d46384e6d8a650994c1b6adcb in term 0.
I20260811 03:04:28.183979 29801 leader_election.cc:302] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcfe39d86d56402e9327f5329c612a61, c38fae8d46384e6d8a650994c1b6adcb; no voters:
I20260811 03:04:28.184332 29968 raft_consensus.cc:2781] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:28.184456 29968 raft_consensus.cc:482] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:28.184612 29968 raft_consensus.cc:3037] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:28.187002 29968 raft_consensus.cc:504] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.187757 29968 leader_election.cc:288] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [CANDIDATE]: Term 1 election: Requested vote from peers bcfe39d86d56402e9327f5329c612a61 (127.27.109.1:39453), 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959)
I20260811 03:04:28.188330 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c5605491b04055b8b52f8707294304" candidate_uuid: "c38fae8d46384e6d8a650994c1b6adcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcfe39d86d56402e9327f5329c612a61"
I20260811 03:04:28.188591 29561 raft_consensus.cc:3037] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:28.188469 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c5605491b04055b8b52f8707294304" candidate_uuid: "c38fae8d46384e6d8a650994c1b6adcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3963c9220ebc450088af555b107fe404"
I20260811 03:04:28.188808 29727 raft_consensus.cc:3037] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:28.191198 29727 raft_consensus.cc:2454] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38fae8d46384e6d8a650994c1b6adcb in term 1.
I20260811 03:04:28.191555 29561 raft_consensus.cc:2454] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38fae8d46384e6d8a650994c1b6adcb in term 1.
I20260811 03:04:28.191737 29803 leader_election.cc:302] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3963c9220ebc450088af555b107fe404, c38fae8d46384e6d8a650994c1b6adcb; no voters:
I20260811 03:04:28.192126 29968 raft_consensus.cc:2781] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:28.192349 29968 raft_consensus.cc:686] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [term 1 LEADER]: Becoming Leader. State: Replica: c38fae8d46384e6d8a650994c1b6adcb, State: Running, Role: LEADER
I20260811 03:04:28.192649 29968 consensus_queue.cc:237] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } }
I20260811 03:04:28.194523 29997 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: TxnStatusTablet state changed. Reason: New leader c38fae8d46384e6d8a650994c1b6adcb. Latest consensus state: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.194905 29997 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:04:28.196556 29376 catalog_manager.cc:5495] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb reported cstate change: term changed from 0 to 1, leader changed from <none> to c38fae8d46384e6d8a650994c1b6adcb (127.27.109.3). New cstate: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } health_report { overall_health: HEALTHY } } }
I20260811 03:04:28.204296 30010 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 03:04:28.204594 30010 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 03:04:28.223006 30004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.215887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:45809 (local address 127.0.0.1:47262)
0811 03:04:28.216204 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:28.216229 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:28.216405 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:28.217175 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:28.217181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:28.217198 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:04:28.217445 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:28.217456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.218852 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.218860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:28.220003 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:28.220016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.220903 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.220910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:28.221113 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.221178 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:04:28.222276 (+ 1098us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:04:28.222303 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 03:04:28.222435 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 03:04:28.222623 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260811 03:04:28.223042 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.215953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.27.109.3:45809)
0811 03:04:28.216754 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:28.216759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:28.216776 (+ 17us) server_negotiation.cc:407] Connection header received
0811 03:04:28.216860 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:28.216864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:28.216949 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:28.217132 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:28.217643 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.218622 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.220162 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.220727 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.220922 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.221470 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:28.222149 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:28.222333 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.222471 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.222562 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":416,"threads_started":1}
I20260811 03:04:28.228570 29727 raft_consensus.cc:1261] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Refusing update from remote peer c38fae8d46384e6d8a650994c1b6adcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:28.229036 29561 raft_consensus.cc:1261] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Refusing update from remote peer c38fae8d46384e6d8a650994c1b6adcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:28.229429 29997 consensus_queue.cc:1035] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:28.230204 29997 consensus_queue.cc:1035] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:28.239768 29967 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: TxnStatusTablet state changed. Reason: New leader c38fae8d46384e6d8a650994c1b6adcb. Latest consensus state: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.240154 29967 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:28.243657 29985 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: TxnStatusTablet state changed. Reason: New leader c38fae8d46384e6d8a650994c1b6adcb. Latest consensus state: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.244060 29985 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:28.245226 30015 mvcc.cc:205] Tried to move back new op lower bound from 7317165949855535104 to 7317165949719891968. Current Snapshot: MvccSnapshot[applied={T|T < 7317165949855535104}]
I20260811 03:04:28.247856 30016 mvcc.cc:205] Tried to move back new op lower bound from 7317165949855535104 to 7317165949719891968. Current Snapshot: MvccSnapshot[applied={T|T < 7317165949855535104}]
I20260811 03:04:28.252137 29967 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.252447 29967 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P 3963c9220ebc450088af555b107fe404: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:28.262861 29997 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.263213 29997 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:04:28.263525 29997 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.263763 29997 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P c38fae8d46384e6d8a650994c1b6adcb: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:04:28.264730 29985 tablet_replica.cc:439] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c38fae8d46384e6d8a650994c1b6adcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 } } peers { permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 } } peers { permanent_uuid: "c38fae8d46384e6d8a650994c1b6adcb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45809 } } }
I20260811 03:04:28.265046 29985 tablet_replica.cc:442] T 61c5605491b04055b8b52f8707294304 P bcfe39d86d56402e9327f5329c612a61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:28.278496 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.271263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.27.109.3:45809)
0811 03:04:28.271506 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:28.271518 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:28.272036 (+ 518us) server_negotiation.cc:407] Connection header received
0811 03:04:28.272237 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:28.272241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:28.272320 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:28.272403 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:28.273058 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.273910 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.276343 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.277000 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.277115 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.277700 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:28.277811 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:28.277898 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.278068 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.278141 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 03:04:28.300172 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.292042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:41979 (local address 127.27.109.3:45809)
0811 03:04:28.292304 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:28.292312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:28.293233 (+ 921us) server_negotiation.cc:407] Connection header received
0811 03:04:28.293368 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:28.293373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:28.293455 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:28.293560 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:28.294321 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.295420 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.298106 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.298576 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.298805 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.299444 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:28.299558 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:28.299632 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.299807 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.299890 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":72}
I20260811 03:04:28.300578 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.292128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:45809 (local address 127.27.109.3:41979)
0811 03:04:28.292999 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:28.293022 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:28.293215 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:28.293616 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:28.293622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:28.293637 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:04:28.294116 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:28.294130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.295635 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.295642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:28.297864 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:28.297884 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.298918 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.298926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:28.299141 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.299229 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:04:28.299671 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:04:28.299685 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 03:04:28.299781 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 03:04:28.299956 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":460,"threads_started":1}
I20260811 03:04:28.313382 29727 raft_consensus.cc:1261] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Refusing update from remote peer c38fae8d46384e6d8a650994c1b6adcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:28.313524 29560 raft_consensus.cc:1261] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Refusing update from remote peer c38fae8d46384e6d8a650994c1b6adcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:28.314081 29997 consensus_queue.cc:1035] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3963c9220ebc450088af555b107fe404" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:28.314389 30023 consensus_queue.cc:1035] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcfe39d86d56402e9327f5329c612a61" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:28.401037 29376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:28.401350 29376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:28.405129 29376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b363d676ce1419ab56e12332da6e7dd: Sending DeleteTablet for 3 replicas of tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6
I20260811 03:04:28.406213 29701 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:28 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40516
I20260811 03:04:28.406426 29540 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:28 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:32848
I20260811 03:04:28.406527 29860 tablet_service.cc:1514] Processing DeleteTablet for tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:28 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:47258
I20260811 03:04:28.416196 30034 tablet_replica.cc:330] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: stopping tablet replica
I20260811 03:04:28.416486 30033 tablet_replica.cc:330] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: stopping tablet replica
I20260811 03:04:28.416805 30033 raft_consensus.cc:2229] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:28.416910 30034 raft_consensus.cc:2229] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:28.417157 30033 raft_consensus.cc:2258] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:28.417222 30034 raft_consensus.cc:2258] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:28.418629 30034 ts_tablet_manager.cc:1905] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:28.418771 30033 ts_tablet_manager.cc:1905] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:28.426968 30034 ts_tablet_manager.cc:1918] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:28.427147 30034 log.cc:1196] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/89fc393c8b2f4dc0a2f5b34ecb4669d6
I20260811 03:04:28.427652 30034 ts_tablet_manager.cc:1939] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P bcfe39d86d56402e9327f5329c612a61: Deleting consensus metadata
I20260811 03:04:28.429023 29359 catalog_manager.cc:4841] TS bcfe39d86d56402e9327f5329c612a61 (127.27.109.1:39453): tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 (table test-table [id=996b010c712c4121b48893925c71ec00]) successfully deleted
I20260811 03:04:28.429165 30033 ts_tablet_manager.cc:1918] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:28.429328 30033 log.cc:1196] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/89fc393c8b2f4dc0a2f5b34ecb4669d6
I20260811 03:04:28.430131 30033 ts_tablet_manager.cc:1939] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P 3963c9220ebc450088af555b107fe404: Deleting consensus metadata
I20260811 03:04:28.431613 29362 catalog_manager.cc:4841] TS 3963c9220ebc450088af555b107fe404 (127.27.109.2:33959): tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 (table test-table [id=996b010c712c4121b48893925c71ec00]) successfully deleted
I20260811 03:04:28.438410 30035 tablet_replica.cc:330] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: stopping tablet replica
I20260811 03:04:28.438715 30035 raft_consensus.cc:2229] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:04:28.439409 30035 raft_consensus.cc:2258] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:28.447676 30035 ts_tablet_manager.cc:1905] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 03:04:28 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417467, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:28.456697 30035 ts_tablet_manager.cc:1918] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:28.456871 30035 log.cc:1196] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/89fc393c8b2f4dc0a2f5b34ecb4669d6
I20260811 03:04:28.457412 30035 ts_tablet_manager.cc:1939] T 89fc393c8b2f4dc0a2f5b34ecb4669d6 P c38fae8d46384e6d8a650994c1b6adcb: Deleting consensus metadata
I20260811 03:04:28.458829 29360 catalog_manager.cc:4841] TS c38fae8d46384e6d8a650994c1b6adcb (127.27.109.3:45809): tablet 89fc393c8b2f4dc0a2f5b34ecb4669d6 (table test-table [id=996b010c712c4121b48893925c71ec00]) successfully deleted
I20260811 03:04:28.465046 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.430493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32870 (local address 127.27.109.1:39453)
0811 03:04:28.443136 (+ 12643us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:28.443143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:28.443168 (+ 25us) server_negotiation.cc:407] Connection header received
0811 03:04:28.443243 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:28.443250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:28.443345 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:28.443484 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:28.445742 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.446992 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.449236 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.449972 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.450105 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.456373 (+ 6268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:28.456401 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:28.456423 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:28.456490 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:28.461193 (+ 4703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.462459 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.462470 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.462474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.462662 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.463164 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.463171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.463174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.463676 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:28.464004 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.464368 (+ 364us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.464623 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12469,"thread_start_us":13119,"threads_started":1}
W20260811 03:04:28.466042 29614 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32870
I20260811 03:04:28.487792 30005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.472087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.27.109.62:39885)
0811 03:04:28.472383 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:28.472390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:28.472605 (+ 215us) server_negotiation.cc:407] Connection header received
0811 03:04:28.472804 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:28.472813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:28.472915 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:28.473005 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:28.474833 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.475856 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.477707 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.478299 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.478447 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.481670 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:28.481676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:28.481679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:28.481720 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:28.484784 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.485673 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.485677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.485680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.485820 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.486269 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.486272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.486274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.486585 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:28.486838 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.487227 (+ 389us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.487438 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260811 03:04:28.488435 29376 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49348
I20260811 03:04:28.490953 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 29461
I20260811 03:04:28.510558 30037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.492794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.27.109.1:39453)
0811 03:04:28.493090 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:28.493098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:28.495239 (+ 2141us) server_negotiation.cc:407] Connection header received
0811 03:04:28.495484 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:28.495488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:28.495607 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:28.495704 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:28.497399 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.498190 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.499861 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.500302 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.500406 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.503738 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:28.503744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:28.503747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:28.503788 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:28.507417 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.508354 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.508359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.508361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.508525 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.509047 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.509051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.509054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.509357 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:28.509791 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.509957 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.510190 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260811 03:04:28.511335 29614 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32884
W20260811 03:04:28.512054 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 29461: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:28.512264 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 29461
I20260811 03:04:28.545665 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 29625
Aug 11 03:04:28 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417467, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:04:28.568226 30042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.546593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.27.109.2:33959)
0811 03:04:28.547331 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:28.547336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:28.547354 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:04:28.547412 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:28.547417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:28.547477 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:28.547576 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:28.549303 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.550185 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.552153 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.552646 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.552785 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.558407 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:28.558427 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:28.558443 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:28.558521 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:28.563303 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.564570 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.564582 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.564586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.564864 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.565442 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.565449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.565452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.566067 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:28.566478 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.567157 (+ 679us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.567444 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":462,"threads_started":1}
W20260811 03:04:28.569103 29772 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40526
W20260811 03:04:28.569823 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 29625: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:28.570034 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 29625
I20260811 03:04:28.601759 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 29782
Aug 11 03:04:28 dist-test-slave-8tdl krb5kdc[29328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417467, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:04:28.622383 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:28.602791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.27.109.3:45809)
0811 03:04:28.603014 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:28.603019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:28.603040 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:04:28.603295 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:28.603299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:28.603384 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:28.603479 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:28.605487 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.606303 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.608356 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:28.608847 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:28.609024 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:28.615225 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:28.615238 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:28.615254 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:28.615296 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:28.618728 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.619583 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.619596 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.619601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.619797 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:28.620204 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:28.620216 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:28.620220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:28.620818 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:28.621182 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:28.621639 (+ 457us) server_negotiation.cc:299] Negotiation successful
0811 03:04:28.621921 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260811 03:04:28.623436 29933 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47288
W20260811 03:04:28.624277 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 29782: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:28.624490 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 29782
I20260811 03:04:28.650914 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 29342
I20260811 03:04:28.652292 29423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:28.772709 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 29342
2026-08-11T03:04:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5517 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:28 dist-test-slave-8tdl krb5kdc[30046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:28 dist-test-slave-8tdl krb5kdc[30046](info): set up 2 sockets
Aug 11 03:04:28 dist-test-slave-8tdl krb5kdc[30046](info): commencing operation
krb5kdc: starting...
W20260811 03:04:29.959913 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:30 dist-test-slave-8tdl krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:04:30.212595 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33375
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:30.577489 30059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:30.577821 30059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:30.577993 30059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:30.585652 30059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:30.585768 30059 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 03:04:30.585850 30059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:30.585971 30059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:30.586064 30059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:30.586184 30059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:30.596935 30059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33375
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:30.598374 30059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:30.599924 30059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:30.612502 30065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:30.612596 30066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:30.612566 30070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:30.613516 30059 server_base.cc:1034] running on GCE node
I20260811 03:04:30.614436 30059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:30.616158 30059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:30.617470 30059 hybrid_clock.cc:648] HybridClock initialized: now 1786417470617397 us; error 39 us; skew 500 ppm
I20260811 03:04:30.617872 30059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:30 dist-test-slave-8tdl krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417470, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:30.621438 30059 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:30.623090 30059 webserver.cc:460] Webserver started at http://127.27.109.62:38647/ using document root <none> and password file <none>
I20260811 03:04:30.623853 30059 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:30.623993 30059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:30.624325 30059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:30.627993 30059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "1a856997b14c4293952cc8dc2a620c52"
format_stamp: "Formatted at 2026-08-11 03:04:30 on dist-test-slave-8tdl"
server_key: "7062ad8d36cfb127a1f2027b93ca012a"
server_key_iv: "611e9f453b139e8894d8bccbc436c3e4"
server_key_version: "encryptionkey@0"
I20260811 03:04:30.628841 30059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "1a856997b14c4293952cc8dc2a620c52"
format_stamp: "Formatted at 2026-08-11 03:04:30 on dist-test-slave-8tdl"
server_key: "7062ad8d36cfb127a1f2027b93ca012a"
server_key_iv: "611e9f453b139e8894d8bccbc436c3e4"
server_key_version: "encryptionkey@0"
I20260811 03:04:30.634472 30059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 03:04:30.638315 30079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:30.639870 30059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:04:30.640130 30059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "1a856997b14c4293952cc8dc2a620c52"
format_stamp: "Formatted at 2026-08-11 03:04:30 on dist-test-slave-8tdl"
server_key: "7062ad8d36cfb127a1f2027b93ca012a"
server_key_iv: "611e9f453b139e8894d8bccbc436c3e4"
server_key_version: "encryptionkey@0"
I20260811 03:04:30.640370 30059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:30.673350 30059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:30.678287 30059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:30.678602 30059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:30.702375 30059 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:33375
I20260811 03:04:30.702395 30150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:33375 every 8 connection(s)
I20260811 03:04:30.703912 30059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:30.707875 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 30059
I20260811 03:04:30.708107 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:30.708297 30151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:30.709000 28084 external_mini_cluster.cc:1428] Setting key 5a4887a71ce59b0d8bd82851b9e02b00
I20260811 03:04:30.724874 30151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: Bootstrap starting.
Aug 11 03:04:30 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:30.729167 30151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:30.730648 30151 log.cc:824] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:30.734485 30151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: No bootstrap required, opened a new log
I20260811 03:04:30.736116 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:30.711415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.27.109.62:33375)
0811 03:04:30.712299 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:30.712312 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:30.712355 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:04:30.714037 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:30.714061 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:30.714533 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:30.715011 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:30.717055 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:30.718455 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:30.720285 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:30.721359 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:30.721546 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:30.726665 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:30.726701 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:30.726724 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:30.726780 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:30.730922 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:30.732063 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:30.732072 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:30.732079 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:30.732296 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:30.732727 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:30.732732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:30.732735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:30.733454 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:30.733755 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:30.734211 (+ 456us) server_negotiation.cc:299] Negotiation successful
0811 03:04:30.734707 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":443,"threads_started":1}
I20260811 03:04:30.743211 30151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a856997b14c4293952cc8dc2a620c52" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33375 } }
I20260811 03:04:30.743659 30151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:30.743779 30151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a856997b14c4293952cc8dc2a620c52, State: Initialized, Role: FOLLOWER
I20260811 03:04:30.744485 30151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a856997b14c4293952cc8dc2a620c52" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33375 } }
I20260811 03:04:30.744787 30151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:30.744936 30151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:30.745255 30151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:30.747598 30151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a856997b14c4293952cc8dc2a620c52" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33375 } }
I20260811 03:04:30.748098 30151 leader_election.cc:302] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a856997b14c4293952cc8dc2a620c52; no voters:
I20260811 03:04:30.748687 30151 leader_election.cc:288] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:30.748850 30157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:30.750010 30157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [term 1 LEADER]: Becoming Leader. State: Replica: 1a856997b14c4293952cc8dc2a620c52, State: Running, Role: LEADER
I20260811 03:04:30.750721 30157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a856997b14c4293952cc8dc2a620c52" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33375 } }
I20260811 03:04:30.751143 30151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:30.754179 30158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a856997b14c4293952cc8dc2a620c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a856997b14c4293952cc8dc2a620c52" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33375 } } }
I20260811 03:04:30.754657 30158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:30.754179 30159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a856997b14c4293952cc8dc2a620c52. Latest consensus state: current_term: 1 leader_uuid: "1a856997b14c4293952cc8dc2a620c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a856997b14c4293952cc8dc2a620c52" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 33375 } } }
I20260811 03:04:30.755782 30159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52 [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:30.757790 30167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:30.763200 30167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:30.773554 30167 catalog_manager.cc:1347] Generated new cluster ID: 01b3e91ab9d14fac880fecefd512af9c
I20260811 03:04:30.773690 30167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:30.795684 30167 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:30.797701 30167 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:30.809540 30167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: Generated new TSK 0
I20260811 03:04:30.810477 30167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:04:30.882725 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33375
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:31.244450 30181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:31.244779 30181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:31.244961 30181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:31.252899 30181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:31.253036 30181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:31.253221 30181 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 03:04:31.253284 30181 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:04:31.253419 30181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:31.265578 30181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:33375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:31.267207 30181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:31.268642 30181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:31.283641 30187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:31.283586 30189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:31.283744 30186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:31.284495 30181 server_base.cc:1034] running on GCE node
I20260811 03:04:31.285223 30181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:31.286347 30181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:31.287546 30181 hybrid_clock.cc:648] HybridClock initialized: now 1786417471287493 us; error 53 us; skew 500 ppm
I20260811 03:04:31.287914 30181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:31 dist-test-slave-8tdl krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417471, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:31.291654 30181 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:04:31.293483 30181 webserver.cc:460] Webserver started at http://127.27.109.1:40959/ using document root <none> and password file <none>
I20260811 03:04:31.294296 30181 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:31.294463 30181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:31.294898 30181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:31.298590 30181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "d5dcda017ad74e24b3c3c140b81a9c57"
format_stamp: "Formatted at 2026-08-11 03:04:31 on dist-test-slave-8tdl"
server_key: "8f7ef4dc57255b7a31bef485d1c397f0"
server_key_iv: "eb7c145baa3e64752ce9c8aa98e82415"
server_key_version: "encryptionkey@0"
I20260811 03:04:31.299543 30181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "d5dcda017ad74e24b3c3c140b81a9c57"
format_stamp: "Formatted at 2026-08-11 03:04:31 on dist-test-slave-8tdl"
server_key: "8f7ef4dc57255b7a31bef485d1c397f0"
server_key_iv: "eb7c145baa3e64752ce9c8aa98e82415"
server_key_version: "encryptionkey@0"
I20260811 03:04:31.305704 30181 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:04:31.310624 30196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:31.312412 30181 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260811 03:04:31.312641 30181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "d5dcda017ad74e24b3c3c140b81a9c57"
format_stamp: "Formatted at 2026-08-11 03:04:31 on dist-test-slave-8tdl"
server_key: "8f7ef4dc57255b7a31bef485d1c397f0"
server_key_iv: "eb7c145baa3e64752ce9c8aa98e82415"
server_key_version: "encryptionkey@0"
I20260811 03:04:31.312894 30181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:31.331368 30181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:31.336107 30181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:31.336476 30181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:31.339903 30181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:31.340054 30181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:31.340221 30181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:31.340322 30181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:04:31 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417471, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:31.368970 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:31.340170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:55103 (local address 127.27.109.62:33375)
0811 03:04:31.340786 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:31.340794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:31.342147 (+ 1353us) server_negotiation.cc:407] Connection header received
0811 03:04:31.344906 (+ 2759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:31.344911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:31.345021 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:31.345703 (+ 682us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:31.348464 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.349554 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.353323 (+ 3769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.353999 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.354179 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:31.360446 (+ 6267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:31.360458 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:31.360466 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:31.360544 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:31.364385 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:31.365825 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:31.365833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:31.365837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:31.366046 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:31.366749 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:31.366755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:31.366758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:31.367175 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:31.367343 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:31.368296 (+ 953us) server_negotiation.cc:299] Negotiation successful
0811 03:04:31.368561 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":299,"threads_started":1}
I20260811 03:04:31.370440 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:31.340986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33375 (local address 127.27.109.1:55103)
0811 03:04:31.341924 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:31.341977 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:31.344494 (+ 2517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:31.345515 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:31.345528 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:31.346758 (+ 1230us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:31.348154 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:31.348204 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.349833 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.349846 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:31.353049 (+ 3203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:31.353084 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.354393 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.354404 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:31.354905 (+ 501us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:31.356014 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:31.356083 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:31.359982 (+ 3899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:31.364618 (+ 4636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:31.364633 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:31.364662 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:31.365601 (+ 939us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:31.366197 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:31.366206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:31.366211 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:31.366572 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:31.367412 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:31.367419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:31.367855 (+ 436us) client_negotiation.cc:769] Sending connection context
0811 03:04:31.368173 (+ 318us) client_negotiation.cc:240] Negotiation successful
0811 03:04:31.368860 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":395,"threads_started":1}
I20260811 03:04:31.391753 30181 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:41325
I20260811 03:04:31.391752 30337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:41325 every 8 connection(s)
I20260811 03:04:31.393916 30181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:31.396831 30338 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33375
I20260811 03:04:31.397104 30338 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:31.397500 30305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:31.378070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33375 (local address 127.27.109.1:37709)
0811 03:04:31.379077 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:31.379103 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:31.379370 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:31.380110 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:31.380115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:31.380835 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:31.381371 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:31.381390 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.382969 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.382975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:31.385505 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:31.385538 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.386816 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.386828 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:31.387052 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:31.387875 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:31.387907 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:31.388916 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:31.393799 (+ 4883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:31.393803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:31.393806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:31.394329 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:31.395056 (+ 727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:31.395061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:31.395064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:31.395273 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:31.396015 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:31.396020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:31.396293 (+ 273us) client_negotiation.cc:769] Sending connection context
0811 03:04:31.396467 (+ 174us) client_negotiation.cc:240] Negotiation successful
0811 03:04:31.396872 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":616,"threads_started":1}
I20260811 03:04:31.397593 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:31.378004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:37709 (local address 127.27.109.62:33375)
0811 03:04:31.378261 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:31.378270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:31.379218 (+ 948us) server_negotiation.cc:407] Connection header received
0811 03:04:31.379615 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:31.379624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:31.379735 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:31.379858 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:31.381635 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.382727 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.385751 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.386324 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.386476 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:31.389177 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:31.389188 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:31.389196 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:31.389259 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:31.393613 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:31.394515 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:31.394525 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:31.394531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:31.394772 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:31.395458 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:31.395465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:31.395468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:31.395860 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:31.396077 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:31.396950 (+ 873us) server_negotiation.cc:299] Negotiation successful
0811 03:04:31.397252 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260811 03:04:31.398087 30338 heartbeater.cc:511] Master 127.27.109.62:33375 requested a full tablet report, sending...
I20260811 03:04:31.400287 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 30181
I20260811 03:04:31.400352 30096 ts_manager.cc:194] Registered new tserver with Master: d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325)
I20260811 03:04:31.400511 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:31.401365 28084 external_mini_cluster.cc:1428] Setting key a554def67d0f71501b94deaffbe9bdda
I20260811 03:04:31.402206 30096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:55103
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:04:31.455263 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33375
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:31.831650 30343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:31.832028 30343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:31.832219 30343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:31.839922 30343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:31.840106 30343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:31.840255 30343 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 03:04:31.840344 30343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:04:31.840488 30343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:04:31.851732 30343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:33375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:31.853310 30343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:31.854874 30343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:31.870142 30349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:31.870177 30352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:31.870146 30350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:31.871387 30343 server_base.cc:1034] running on GCE node
I20260811 03:04:31.872040 30343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:31.873184 30343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:31.874444 30343 hybrid_clock.cc:648] HybridClock initialized: now 1786417471874385 us; error 51 us; skew 500 ppm
I20260811 03:04:31.874915 30343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:31 dist-test-slave-8tdl krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417471, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:31.878422 30343 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:04:31.880344 30343 webserver.cc:460] Webserver started at http://127.27.109.2:40609/ using document root <none> and password file <none>
I20260811 03:04:31.881135 30343 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:31.881294 30343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:31.881639 30343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:31.884862 30343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "bf1e8f4327d44c4393b5dcacb70f1660"
format_stamp: "Formatted at 2026-08-11 03:04:31 on dist-test-slave-8tdl"
server_key: "b3dcebd3644c06bfa32dda405c0a7d12"
server_key_iv: "0de158799e3647032a96b7b5a1486915"
server_key_version: "encryptionkey@0"
I20260811 03:04:31.885718 30343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "bf1e8f4327d44c4393b5dcacb70f1660"
format_stamp: "Formatted at 2026-08-11 03:04:31 on dist-test-slave-8tdl"
server_key: "b3dcebd3644c06bfa32dda405c0a7d12"
server_key_iv: "0de158799e3647032a96b7b5a1486915"
server_key_version: "encryptionkey@0"
I20260811 03:04:31.891673 30343 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 03:04:31.896293 30361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:31.898183 30343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 03:04:31.898401 30343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "bf1e8f4327d44c4393b5dcacb70f1660"
format_stamp: "Formatted at 2026-08-11 03:04:31 on dist-test-slave-8tdl"
server_key: "b3dcebd3644c06bfa32dda405c0a7d12"
server_key_iv: "0de158799e3647032a96b7b5a1486915"
server_key_version: "encryptionkey@0"
I20260811 03:04:31.898651 30343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:31.917572 30343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:31.922878 30343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:31.923257 30343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:31.926561 30343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:31.926744 30343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:31.926931 30343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:31.927111 30343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:04:31 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417471, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:31.954342 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:31.927175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39729 (local address 127.27.109.62:33375)
0811 03:04:31.927705 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:31.927712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:31.929100 (+ 1388us) server_negotiation.cc:407] Connection header received
0811 03:04:31.931804 (+ 2704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:31.931812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:31.931936 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:31.932113 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:31.934761 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.936358 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.939246 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.939721 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.939868 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:31.945889 (+ 6021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:31.945895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:31.945906 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:31.945948 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:31.949295 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:31.950432 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:31.950437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:31.950439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:31.950624 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:31.951488 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:31.951497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:31.951502 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:31.951866 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:31.952046 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:31.953230 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0811 03:04:31.953665 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":278,"threads_started":1}
I20260811 03:04:31.954950 30373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:31.927912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33375 (local address 127.27.109.2:39729)
0811 03:04:31.928873 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:31.928929 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:31.931356 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:31.932442 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:31.932455 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:31.933384 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:31.934443 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:31.934491 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.936591 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.936603 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:31.939068 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:31.939084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.939996 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.940005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:31.940369 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:31.941361 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:31.941413 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:31.945607 (+ 4194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:31.949468 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:31.949479 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:31.949505 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:31.950272 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:31.950870 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:31.950880 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:31.950884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:31.951318 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:31.952025 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:31.952034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:31.952665 (+ 631us) client_negotiation.cc:769] Sending connection context
0811 03:04:31.953038 (+ 373us) client_negotiation.cc:240] Negotiation successful
0811 03:04:31.953607 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":414,"threads_started":1}
I20260811 03:04:31.984149 30370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:31.965398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:57855 (local address 127.27.109.62:33375)
0811 03:04:31.965658 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:31.965687 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:31.966116 (+ 429us) server_negotiation.cc:407] Connection header received
0811 03:04:31.966542 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:31.966551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:31.966660 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:31.966800 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:31.969251 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.970077 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.973381 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.974081 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.974262 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:31.976876 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:31.976884 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:31.976889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:31.976941 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:31.980906 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:31.981725 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:31.981731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:31.981735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:31.981875 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:31.982287 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:31.982293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:31.982296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:31.982660 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:31.982894 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:31.983506 (+ 612us) server_negotiation.cc:299] Negotiation successful
0811 03:04:31.983770 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260811 03:04:31.984309 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:31.964778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33375 (local address 127.27.109.2:57855)
0811 03:04:31.965931 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:31.965960 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:31.966290 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:31.967115 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:31.967123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:31.968213 (+ 1090us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:31.969042 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:31.969064 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.970387 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.970397 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:31.973148 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:31.973180 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:31.974361 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:31.974369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:31.974566 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:31.975518 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:31.975542 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:31.976521 (+ 979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:31.981083 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:31.981089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:31.981093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:31.981575 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:31.982009 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:31.982015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:31.982019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:31.982168 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:31.982976 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:31.982988 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:31.983216 (+ 228us) client_negotiation.cc:769] Sending connection context
0811 03:04:31.983398 (+ 182us) client_negotiation.cc:240] Negotiation successful
0811 03:04:31.983805 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":908,"threads_started":1}
I20260811 03:04:31.991585 30343 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:44017
I20260811 03:04:31.991655 30500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:44017 every 8 connection(s)
I20260811 03:04:31.993166 30343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:04:31.995923 30501 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33375
I20260811 03:04:31.996269 30501 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:31.997102 30501 heartbeater.cc:511] Master 127.27.109.62:33375 requested a full tablet report, sending...
I20260811 03:04:31.998983 30095 ts_manager.cc:194] Registered new tserver with Master: bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017)
I20260811 03:04:32.000057 30095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:39729
I20260811 03:04:32.001626 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 30343
I20260811 03:04:32.001847 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:04:32.002375 28084 external_mini_cluster.cc:1428] Setting key 99f6c1f94e662c958907f06a76205738
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:04:32.060359 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:33375
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:04:32.405475 30338 heartbeater.cc:503] Master 127.27.109.62:33375 was elected leader, sending a full tablet report...
W20260811 03:04:32.434705 30505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:32.435058 30505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:32.435268 30505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:32.442866 30505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:32.443030 30505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:32.443204 30505 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 03:04:32.443287 30505 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:04:32.443414 30505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:04:32.454376 30505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:33375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:32.455876 30505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:32.457557 30505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:32.473137 30513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:32.473049 30511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:32.473210 30510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:32.478854 30505 server_base.cc:1034] running on GCE node
I20260811 03:04:32.479735 30505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:32.481086 30505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:32.482317 30505 hybrid_clock.cc:648] HybridClock initialized: now 1786417472482263 us; error 54 us; skew 500 ppm
I20260811 03:04:32.482761 30505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:32 dist-test-slave-8tdl krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417472, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:32.486462 30505 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:04:32.488345 30505 webserver.cc:460] Webserver started at http://127.27.109.3:39763/ using document root <none> and password file <none>
I20260811 03:04:32.489171 30505 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:32.489313 30505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:32.489748 30505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:32.493037 30505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "f49c98ba307b4295a7cb7bff38c44f2b"
format_stamp: "Formatted at 2026-08-11 03:04:32 on dist-test-slave-8tdl"
server_key: "6f06d9e111e7ad89d628cf2b9a898ff4"
server_key_iv: "56d8dad29207cff7dad1be90f4a1f0b7"
server_key_version: "encryptionkey@0"
I20260811 03:04:32.493818 30505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "f49c98ba307b4295a7cb7bff38c44f2b"
format_stamp: "Formatted at 2026-08-11 03:04:32 on dist-test-slave-8tdl"
server_key: "6f06d9e111e7ad89d628cf2b9a898ff4"
server_key_iv: "56d8dad29207cff7dad1be90f4a1f0b7"
server_key_version: "encryptionkey@0"
I20260811 03:04:32.499992 30505 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260811 03:04:32.504130 30521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:32.505843 30505 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 03:04:32.506069 30505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "f49c98ba307b4295a7cb7bff38c44f2b"
format_stamp: "Formatted at 2026-08-11 03:04:32 on dist-test-slave-8tdl"
server_key: "6f06d9e111e7ad89d628cf2b9a898ff4"
server_key_iv: "56d8dad29207cff7dad1be90f4a1f0b7"
server_key_version: "encryptionkey@0"
I20260811 03:04:32.506301 30505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:32.528820 30505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:32.533806 30505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:32.534171 30505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:32.537477 30505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:32.537657 30505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:32.537796 30505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:32.537911 30505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:04:32 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417472, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:32.567253 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.538173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:35633 (local address 127.27.109.62:33375)
0811 03:04:32.538956 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:32.538963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:32.540290 (+ 1327us) server_negotiation.cc:407] Connection header received
0811 03:04:32.542826 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:32.542832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:32.543027 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:32.543171 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:32.546568 (+ 3397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.547755 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.551641 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.552469 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.552619 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.558590 (+ 5971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:32.558596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:32.558620 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:32.558680 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:32.562786 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:32.564090 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:32.564097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:32.564103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:32.564293 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:32.565039 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:32.565043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:32.565045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:32.565352 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:32.565524 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:32.566494 (+ 970us) server_negotiation.cc:299] Negotiation successful
0811 03:04:32.566802 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":525,"threads_started":1}
I20260811 03:04:32.568531 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.539099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33375 (local address 127.27.109.3:35633)
0811 03:04:32.540071 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:32.540122 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:32.542262 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:32.543627 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:32.543644 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:32.544885 (+ 1241us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:32.546250 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.546293 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.548090 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.548099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.551348 (+ 3249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.551378 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.552742 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.552747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.552973 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.553987 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:32.554020 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:32.558209 (+ 4189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:32.563020 (+ 4811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:32.563031 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:32.563056 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:32.563837 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:32.564470 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:32.564474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:32.564477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:32.564861 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:32.565526 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:32.565536 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:32.565987 (+ 451us) client_negotiation.cc:769] Sending connection context
0811 03:04:32.566372 (+ 385us) client_negotiation.cc:240] Negotiation successful
0811 03:04:32.567116 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":407,"threads_started":1}
I20260811 03:04:32.594416 30505 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38069
I20260811 03:04:32.594416 30659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38069 every 8 connection(s)
I20260811 03:04:32.596191 30505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:04:32.598762 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.578966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:56453 (local address 127.27.109.62:33375)
0811 03:04:32.579197 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:32.579205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:32.580170 (+ 965us) server_negotiation.cc:407] Connection header received
0811 03:04:32.580440 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:32.580450 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:32.580555 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:32.580680 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:32.582713 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.583976 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.586395 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.587207 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.587360 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.591037 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:32.591046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:32.591066 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:32.591105 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:32.594831 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:32.595557 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:32.595563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:32.595569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:32.595750 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:32.596136 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:32.596141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:32.596147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:32.596559 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:32.596734 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:32.597381 (+ 647us) server_negotiation.cc:299] Negotiation successful
0811 03:04:32.597729 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 03:04:32.599419 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 30505
I20260811 03:04:32.599582 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:04:32.600001 28084 external_mini_cluster.cc:1428] Setting key 452cf3cb3bcd87a3fc02e501b0a3a5de
I20260811 03:04:32.598232 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.579067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33375 (local address 127.27.109.3:56453)
0811 03:04:32.580012 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:32.580041 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:32.580247 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:32.580939 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:32.580947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:32.581874 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:32.582478 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.582497 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.584145 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.584151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.586205 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.586228 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.587562 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.587578 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.587862 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.588822 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:32.588856 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:32.590656 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:32.594988 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:32.594992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:32.594994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:32.595424 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:32.595873 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:32.595877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:32.595879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:32.596003 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:32.596691 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:32.596694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:32.596977 (+ 283us) client_negotiation.cc:769] Sending connection context
0811 03:04:32.597205 (+ 228us) client_negotiation.cc:240] Negotiation successful
0811 03:04:32.597635 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":533,"threads_started":1}
I20260811 03:04:32.598802 30660 heartbeater.cc:347] Connected to a master server at 127.27.109.62:33375
I20260811 03:04:32.601156 30660 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:32.601966 30660 heartbeater.cc:511] Master 127.27.109.62:33375 requested a full tablet report, sending...
I20260811 03:04:32.603570 30095 ts_manager.cc:194] Registered new tserver with Master: f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069)
I20260811 03:04:32.604682 30095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:35633
I20260811 03:04:32.605453 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 03:04:32 dist-test-slave-8tdl krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 03:04:32 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:32.665853 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.649597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.27.109.62:33375)
0811 03:04:32.649840 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:32.649844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:32.649900 (+ 56us) server_negotiation.cc:407] Connection header received
0811 03:04:32.650074 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:32.650078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:32.650164 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:32.650255 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:32.651989 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.652966 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.654344 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.654947 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.655063 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.659692 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:32.659699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:32.659703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:32.659752 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:32.662957 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:32.663747 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:32.663752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:32.663754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:32.663905 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:32.664414 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:32.664417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:32.664419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:32.664741 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:32.665043 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:32.665273 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 03:04:32.665505 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260811 03:04:32.670814 30095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:04:32.678553 30095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:32.716439 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.692698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41325 (local address 127.0.0.1:44814)
0811 03:04:32.693576 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:32.693633 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:32.693822 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:32.707623 (+ 13801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:32.707631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:32.707678 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:32.708092 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.708116 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.710030 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.710035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.712928 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.712949 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.715297 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.715305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.715573 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.715632 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 03:04:32.715758 (+ 126us) client_negotiation.cc:240] Negotiation successful
0811 03:04:32.716016 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":3,"thread_start_us":434,"threads_started":1}
I20260811 03:04:32.717373 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.692889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.27.109.1:41325)
0811 03:04:32.706768 (+ 13879us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:32.706781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:32.706820 (+ 39us) server_negotiation.cc:407] Connection header received
0811 03:04:32.706967 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:32.706979 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:32.707328 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:32.707587 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:32.708336 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.709858 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.713171 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.714731 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.714899 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.715127 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:32.716812 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0811 03:04:32.717077 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13482,"thread_start_us":13337,"threads_started":1}
I20260811 03:04:32.719671 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.693137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44017 (local address 127.0.0.1:54060)
0811 03:04:32.705737 (+ 12600us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:32.705760 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:32.705920 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:32.709270 (+ 3350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:32.709275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:32.709304 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:32.709931 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.709949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.712883 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.712889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.715359 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.715374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.718694 (+ 3320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.718705 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.718882 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.718916 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:04:32.719003 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 03:04:32.719167 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12433,"mutex_wait_us":1,"thread_start_us":19419,"threads_started":1}
I20260811 03:04:32.720589 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.693681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.27.109.2:44017)
0811 03:04:32.708275 (+ 14594us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:32.708288 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:32.708324 (+ 36us) server_negotiation.cc:407] Connection header received
0811 03:04:32.708440 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:32.708451 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:32.708746 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:32.709001 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:32.710183 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.712232 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.715589 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.717070 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.717212 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.717431 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:32.719971 (+ 2540us) server_negotiation.cc:299] Negotiation successful
0811 03:04:32.720159 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14406,"thread_start_us":14295,"threads_started":1}
I20260811 03:04:32.721719 30256 tablet_service.cc:1467] Processing CreateTablet for tablet dad2956c249c48cba85edeb9773a2861 (DEFAULT_TABLE table=test-table [id=ffc49638ea4d47acab04c30cd062b052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:32.722735 30680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.694186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38069 (local address 127.0.0.1:57296)
0811 03:04:32.712890 (+ 18704us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:32.712920 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:32.713077 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:32.714219 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:32.714224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:32.714250 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:32.714706 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.714718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.716883 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.716888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.719238 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.719253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.721304 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.721312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.721495 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.722120 (+ 625us) client_negotiation.cc:769] Sending connection context
0811 03:04:32.722203 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 03:04:32.722340 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18543,"thread_start_us":11220,"threads_started":1}
I20260811 03:04:32.723352 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad2956c249c48cba85edeb9773a2861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:32.723502 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.694714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.27.109.3:38069)
0811 03:04:32.705518 (+ 10804us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:32.705529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:32.713232 (+ 7703us) server_negotiation.cc:407] Connection header received
0811 03:04:32.713355 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:32.713362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:32.713673 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:32.713885 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:32.714974 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.716715 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.719489 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.721058 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.721797 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.722044 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:32.722972 (+ 928us) server_negotiation.cc:299] Negotiation successful
0811 03:04:32.723148 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10535,"thread_start_us":10270,"threads_started":1}
I20260811 03:04:32.724460 30421 tablet_service.cc:1467] Processing CreateTablet for tablet dad2956c249c48cba85edeb9773a2861 (DEFAULT_TABLE table=test-table [id=ffc49638ea4d47acab04c30cd062b052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:32.725723 30421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad2956c249c48cba85edeb9773a2861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:32.727490 30580 tablet_service.cc:1467] Processing CreateTablet for tablet dad2956c249c48cba85edeb9773a2861 (DEFAULT_TABLE table=test-table [id=ffc49638ea4d47acab04c30cd062b052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:32.730363 30580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad2956c249c48cba85edeb9773a2861. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:32.738965 30685 tablet_bootstrap.cc:492] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Bootstrap starting.
I20260811 03:04:32.744199 30685 tablet_bootstrap.cc:654] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:32.744195 30686 tablet_bootstrap.cc:492] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Bootstrap starting.
I20260811 03:04:32.746491 30685 log.cc:824] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:32.749596 30686 tablet_bootstrap.cc:654] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:32.751323 30686 log.cc:824] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:32.751341 30687 tablet_bootstrap.cc:492] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Bootstrap starting.
I20260811 03:04:32.755138 30685 tablet_bootstrap.cc:492] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: No bootstrap required, opened a new log
I20260811 03:04:32.755136 30686 tablet_bootstrap.cc:492] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: No bootstrap required, opened a new log
I20260811 03:04:32.755313 30687 tablet_bootstrap.cc:654] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:32.755457 30686 ts_tablet_manager.cc:1397] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260811 03:04:32.755477 30685 ts_tablet_manager.cc:1397] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20260811 03:04:32.756764 30687 log.cc:824] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:32.760058 30687 tablet_bootstrap.cc:492] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: No bootstrap required, opened a new log
I20260811 03:04:32.760349 30687 ts_tablet_manager.cc:1397] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260811 03:04:32.767146 30686 raft_consensus.cc:348] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.767146 30685 raft_consensus.cc:348] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.767731 30686 raft_consensus.cc:374] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:32.767737 30685 raft_consensus.cc:374] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:32.767865 30686 raft_consensus.cc:729] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1e8f4327d44c4393b5dcacb70f1660, State: Initialized, Role: FOLLOWER
I20260811 03:04:32.767855 30685 raft_consensus.cc:729] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5dcda017ad74e24b3c3c140b81a9c57, State: Initialized, Role: FOLLOWER
I20260811 03:04:32.768105 30687 raft_consensus.cc:348] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.768610 30687 raft_consensus.cc:374] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:32.768735 30687 raft_consensus.cc:729] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49c98ba307b4295a7cb7bff38c44f2b, State: Initialized, Role: FOLLOWER
I20260811 03:04:32.768604 30685 consensus_queue.cc:260] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.768604 30686 consensus_queue.cc:260] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.769371 30687 consensus_queue.cc:260] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.772085 30501 heartbeater.cc:503] Master 127.27.109.62:33375 was elected leader, sending a full tablet report...
I20260811 03:04:32.772936 30686 ts_tablet_manager.cc:1428] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260811 03:04:32.773301 30660 heartbeater.cc:503] Master 127.27.109.62:33375 was elected leader, sending a full tablet report...
I20260811 03:04:32.773602 30687 ts_tablet_manager.cc:1428] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260811 03:04:32.774901 30685 ts_tablet_manager.cc:1428] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
W20260811 03:04:32.848870 30661 tablet.cc:2369] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:04:32.898015 30339 tablet.cc:2369] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:04:32.918355 30694 raft_consensus.cc:482] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:32.918715 30694 raft_consensus.cc:504] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.920483 30694 leader_election.cc:288] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325), bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017)
I20260811 03:04:32.926738 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.921055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:53893 (local address 127.27.109.2:44017)
0811 03:04:32.921250 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:32.921257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:32.921334 (+ 77us) server_negotiation.cc:407] Connection header received
0811 03:04:32.921588 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:32.921592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:32.921654 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:32.921759 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:32.922287 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.923127 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.924894 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.925660 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.925838 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.925916 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:32.926288 (+ 372us) server_negotiation.cc:299] Negotiation successful
0811 03:04:32.926420 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 03:04:32.926738 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.920965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44017 (local address 127.27.109.3:53893)
0811 03:04:32.921221 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:32.921243 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:32.921409 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:32.921785 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:32.921795 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:32.921834 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:32.922063 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.922072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.923285 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.923296 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.924724 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.924737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.925830 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.925836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.926122 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.926149 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 03:04:32.926259 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 03:04:32.926400 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":60}
I20260811 03:04:32.927772 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad2956c249c48cba85edeb9773a2861" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" is_pre_election: true
I20260811 03:04:32.928383 30449 raft_consensus.cc:2454] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 0.
I20260811 03:04:32.928838 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.921108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:38621 (local address 127.27.109.1:41325)
0811 03:04:32.921329 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:32.921334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:32.922555 (+ 1221us) server_negotiation.cc:407] Connection header received
0811 03:04:32.922838 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:32.922842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:32.922919 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:32.923036 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:32.923863 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.924703 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.926555 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.927775 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.928005 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.928093 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:32.928427 (+ 334us) server_negotiation.cc:299] Negotiation successful
0811 03:04:32.928569 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 03:04:32.928999 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:32.921545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41325 (local address 127.27.109.3:38621)
0811 03:04:32.922436 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:32.922453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:32.922619 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:32.923056 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:32.923061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:32.923090 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:32.923663 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.923677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.924870 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.924876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.926374 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:32.926387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:32.928048 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:32.928056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:32.928246 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:32.928272 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 03:04:32.928403 (+ 131us) client_negotiation.cc:240] Negotiation successful
0811 03:04:32.928548 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":1,"thread_start_us":508,"threads_started":1}
I20260811 03:04:32.929188 30524 leader_election.cc:302] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf1e8f4327d44c4393b5dcacb70f1660, f49c98ba307b4295a7cb7bff38c44f2b; no voters:
I20260811 03:04:32.929646 30694 raft_consensus.cc:2781] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:32.929800 30694 raft_consensus.cc:482] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:32.929921 30694 raft_consensus.cc:3037] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:32.930240 30281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad2956c249c48cba85edeb9773a2861" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" is_pre_election: true
I20260811 03:04:32.930860 30281 raft_consensus.cc:2454] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 0.
I20260811 03:04:32.932717 30694 raft_consensus.cc:504] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.933444 30694 leader_election.cc:288] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 election: Requested vote from peers d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325), bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017)
I20260811 03:04:32.934029 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad2956c249c48cba85edeb9773a2861" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1e8f4327d44c4393b5dcacb70f1660"
I20260811 03:04:32.934062 30281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad2956c249c48cba85edeb9773a2861" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5dcda017ad74e24b3c3c140b81a9c57"
I20260811 03:04:32.934266 30449 raft_consensus.cc:3037] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:32.934293 30281 raft_consensus.cc:3037] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:32.936872 30449 raft_consensus.cc:2454] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 1.
I20260811 03:04:32.936983 30281 raft_consensus.cc:2454] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 1.
I20260811 03:04:32.937469 30524 leader_election.cc:302] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf1e8f4327d44c4393b5dcacb70f1660, f49c98ba307b4295a7cb7bff38c44f2b; no voters:
I20260811 03:04:32.937819 30694 raft_consensus.cc:2781] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:32.938299 30694 raft_consensus.cc:686] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 LEADER]: Becoming Leader. State: Replica: f49c98ba307b4295a7cb7bff38c44f2b, State: Running, Role: LEADER
I20260811 03:04:32.938863 30694 consensus_queue.cc:237] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } }
I20260811 03:04:32.944191 30095 catalog_manager.cc:5495] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3). New cstate: current_term: 1 leader_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } health_report { overall_health: UNKNOWN } } }
W20260811 03:04:32.995865 30502 tablet.cc:2369] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 03:04:33 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417470, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:33.053589 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.033508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:33375 (local address 127.0.0.1:37054)
0811 03:04:33.034243 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:33.034260 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:33.034490 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:33.034920 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:33.034925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:33.035768 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:33.036063 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.036075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.037238 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.037243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.038347 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.038359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.039285 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.039291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.039491 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.043810 (+ 4319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:33.043831 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:33.046741 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:33.050320 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:33.050334 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:33.050340 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:33.051139 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:33.051694 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:33.051698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:33.051700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:33.051910 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:33.052608 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:33.052614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:33.052811 (+ 197us) client_negotiation.cc:769] Sending connection context
0811 03:04:33.052963 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 03:04:33.053213 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":439,"threads_started":1}
I20260811 03:04:33.053737 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.033491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.27.109.62:33375)
0811 03:04:33.033817 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:33.033827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:33.034364 (+ 537us) server_negotiation.cc:407] Connection header received
0811 03:04:33.034659 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:33.034663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:33.034751 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:33.034881 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:33.036221 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.037075 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.038514 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.039069 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.039228 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.047043 (+ 7815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:33.047050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:33.047053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:33.047092 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:33.050132 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:33.051305 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:33.051310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:33.051313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:33.051554 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:33.052035 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:33.052039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:33.052042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:33.052415 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:33.052616 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:33.053083 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 03:04:33.053355 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":92}
I20260811 03:04:33.059971 30095 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:37054:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 03:04:33.061233 30095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:33.069327 30256 tablet_service.cc:1467] Processing CreateTablet for tablet bfdc59fba65a4a419793d686f3dacf3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d765b5229d8a497bb06d34c3cec63815]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:04:33.069433 30421 tablet_service.cc:1467] Processing CreateTablet for tablet bfdc59fba65a4a419793d686f3dacf3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d765b5229d8a497bb06d34c3cec63815]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:04:33.069650 30580 tablet_service.cc:1467] Processing CreateTablet for tablet bfdc59fba65a4a419793d686f3dacf3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d765b5229d8a497bb06d34c3cec63815]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:04:33.070434 30421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfdc59fba65a4a419793d686f3dacf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:33.070446 30580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfdc59fba65a4a419793d686f3dacf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:33.070499 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfdc59fba65a4a419793d686f3dacf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:33.076730 30686 tablet_bootstrap.cc:492] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: Bootstrap starting.
I20260811 03:04:33.078246 30686 tablet_bootstrap.cc:654] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:33.078992 30685 tablet_bootstrap.cc:492] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: Bootstrap starting.
I20260811 03:04:33.079244 30687 tablet_bootstrap.cc:492] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: Bootstrap starting.
I20260811 03:04:33.081041 30685 tablet_bootstrap.cc:654] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:33.081072 30687 tablet_bootstrap.cc:654] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:33.084252 30685 tablet_bootstrap.cc:492] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: No bootstrap required, opened a new log
I20260811 03:04:33.084434 30685 ts_tablet_manager.cc:1397] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260811 03:04:33.085423 30686 tablet_bootstrap.cc:492] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: No bootstrap required, opened a new log
I20260811 03:04:33.085529 30687 tablet_bootstrap.cc:492] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: No bootstrap required, opened a new log
I20260811 03:04:33.085420 30685 raft_consensus.cc:348] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.085621 30686 ts_tablet_manager.cc:1397] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260811 03:04:33.085695 30687 ts_tablet_manager.cc:1397] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 03:04:33.085714 30685 raft_consensus.cc:374] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:33.085813 30685 raft_consensus.cc:729] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5dcda017ad74e24b3c3c140b81a9c57, State: Initialized, Role: FOLLOWER
I20260811 03:04:33.086089 30685 consensus_queue.cc:260] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.086624 30687 raft_consensus.cc:348] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.086614 30686 raft_consensus.cc:348] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.086896 30687 raft_consensus.cc:374] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:33.086913 30686 raft_consensus.cc:374] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:33.086639 30692 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.087001 30687 raft_consensus.cc:729] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49c98ba307b4295a7cb7bff38c44f2b, State: Initialized, Role: FOLLOWER
I20260811 03:04:33.087088 30692 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:33.087083 30686 raft_consensus.cc:729] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1e8f4327d44c4393b5dcacb70f1660, State: Initialized, Role: FOLLOWER
I20260811 03:04:33.087284 30687 consensus_queue.cc:260] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.087319 30686 consensus_queue.cc:260] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.087924 30687 ts_tablet_manager.cc:1428] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 03:04:33.087976 30686 ts_tablet_manager.cc:1428] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 03:04:33.087862 30693 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.088336 30693 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:33.088707 30694 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.089030 30694 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:33.089207 30685 ts_tablet_manager.cc:1428] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260811 03:04:33.334180 30693 raft_consensus.cc:482] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:33.334436 30693 raft_consensus.cc:504] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.336349 30693 leader_election.cc:288] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325), f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069)
I20260811 03:04:33.346326 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.336830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:43649 (local address 127.27.109.1:41325)
0811 03:04:33.337051 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:33.337058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:33.337768 (+ 710us) server_negotiation.cc:407] Connection header received
0811 03:04:33.338107 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:33.338113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:33.338223 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:33.338373 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:33.339288 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.340328 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.343306 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.344813 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.345056 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.345153 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:33.345812 (+ 659us) server_negotiation.cc:299] Negotiation successful
0811 03:04:33.345987 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 03:04:33.346313 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.338015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:35811 (local address 127.27.109.3:38069)
0811 03:04:33.339160 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:33.339165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:33.339217 (+ 52us) server_negotiation.cc:407] Connection header received
0811 03:04:33.339296 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:33.339301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:33.339394 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:33.339515 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:33.340220 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.341051 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.343710 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.345250 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.345415 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.345481 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:33.345852 (+ 371us) server_negotiation.cc:299] Negotiation successful
0811 03:04:33.346010 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":907,"threads_started":1}
I20260811 03:04:33.346554 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.337822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38069 (local address 127.27.109.2:35811)
0811 03:04:33.338618 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:33.338640 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:33.338798 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:33.339557 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:33.339560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:33.339585 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:33.340038 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.340050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.341208 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.341212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.343520 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.343532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.345472 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.345478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.345629 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.345666 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 03:04:33.345808 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 03:04:33.345974 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":18,"thread_start_us":416,"threads_started":1}
I20260811 03:04:33.346662 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.336809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41325 (local address 127.27.109.2:43649)
0811 03:04:33.337612 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:33.337644 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:33.337811 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:33.338546 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:33.338551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:33.338585 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:33.339091 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.339109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.340491 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.340495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.343005 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.343024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.345341 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.345351 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.345559 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.345602 (+ 43us) client_negotiation.cc:769] Sending connection context
0811 03:04:33.345782 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 03:04:33.345988 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":82,"thread_start_us":468,"threads_started":1}
I20260811 03:04:33.347442 30281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfdc59fba65a4a419793d686f3dacf3f" candidate_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" is_pre_election: true
I20260811 03:04:33.347764 30281 raft_consensus.cc:2454] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1e8f4327d44c4393b5dcacb70f1660 in term 0.
I20260811 03:04:33.347677 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfdc59fba65a4a419793d686f3dacf3f" candidate_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" is_pre_election: true
I20260811 03:04:33.348124 30608 raft_consensus.cc:2454] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1e8f4327d44c4393b5dcacb70f1660 in term 0.
I20260811 03:04:33.348420 30362 leader_election.cc:302] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf1e8f4327d44c4393b5dcacb70f1660, d5dcda017ad74e24b3c3c140b81a9c57; no voters:
I20260811 03:04:33.348975 30693 raft_consensus.cc:2781] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:33.349123 30693 raft_consensus.cc:482] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:33.349253 30693 raft_consensus.cc:3037] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:33.352002 30693 raft_consensus.cc:504] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.352758 30693 leader_election.cc:288] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [CANDIDATE]: Term 1 election: Requested vote from peers d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325), f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069)
I20260811 03:04:33.353459 30281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfdc59fba65a4a419793d686f3dacf3f" candidate_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5dcda017ad74e24b3c3c140b81a9c57"
I20260811 03:04:33.353735 30281 raft_consensus.cc:3037] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:33.353747 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfdc59fba65a4a419793d686f3dacf3f" candidate_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49c98ba307b4295a7cb7bff38c44f2b"
I20260811 03:04:33.354117 30608 raft_consensus.cc:3037] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:33.356137 30281 raft_consensus.cc:2454] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1e8f4327d44c4393b5dcacb70f1660 in term 1.
I20260811 03:04:33.356802 30362 leader_election.cc:302] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf1e8f4327d44c4393b5dcacb70f1660, d5dcda017ad74e24b3c3c140b81a9c57; no voters:
I20260811 03:04:33.356827 30608 raft_consensus.cc:2454] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1e8f4327d44c4393b5dcacb70f1660 in term 1.
I20260811 03:04:33.357203 30693 raft_consensus.cc:2781] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:33.357760 30693 raft_consensus.cc:686] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 LEADER]: Becoming Leader. State: Replica: bf1e8f4327d44c4393b5dcacb70f1660, State: Running, Role: LEADER
I20260811 03:04:33.358484 30693 consensus_queue.cc:237] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.361761 30713 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: TxnStatusTablet state changed. Reason: New leader bf1e8f4327d44c4393b5dcacb70f1660. Latest consensus state: current_term: 1 leader_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.362295 30713 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:04:33.363276 30716 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 03:04:33.363610 30716 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 03:04:33.363332 30094 catalog_manager.cc:5495] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } health_report { overall_health: UNKNOWN } } }
I20260811 03:04:33.367561 30694 consensus_queue.cc:1035] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:33.378553 30694 consensus_queue.cc:1035] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:33.420900 30705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.409335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44017 (local address 127.0.0.1:54070)
0811 03:04:33.410053 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:33.410081 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:33.410284 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:33.411887 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:33.411895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:33.411917 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:04:33.412229 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.412244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.414401 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.414410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.416770 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.416795 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.418474 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.418481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.418765 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.418873 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:04:33.420212 (+ 1339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:04:33.420245 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 03:04:33.420327 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 03:04:33.420535 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414}
I20260811 03:04:33.421744 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.410788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.27.109.2:44017)
0811 03:04:33.411051 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:33.411060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:33.411087 (+ 27us) server_negotiation.cc:407] Connection header received
0811 03:04:33.411225 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:33.411233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:33.411430 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:33.411611 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:33.412514 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.414166 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.417005 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.418072 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.418258 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.419132 (+ 874us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:33.419959 (+ 827us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:33.420146 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:33.421302 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0811 03:04:33.421401 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260811 03:04:33.430157 30281 raft_consensus.cc:1261] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Refusing update from remote peer bf1e8f4327d44c4393b5dcacb70f1660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:33.430500 30608 raft_consensus.cc:1261] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Refusing update from remote peer bf1e8f4327d44c4393b5dcacb70f1660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:33.431123 30713 consensus_queue.cc:1035] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:33.432459 30713 consensus_queue.cc:1035] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:33.443964 30692 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: TxnStatusTablet state changed. Reason: New leader bf1e8f4327d44c4393b5dcacb70f1660. Latest consensus state: current_term: 1 leader_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.444406 30692 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:33.449309 30693 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.449699 30693 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:04:33.450937 30721 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: TxnStatusTablet state changed. Reason: New leader bf1e8f4327d44c4393b5dcacb70f1660. Latest consensus state: current_term: 1 leader_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.451372 30721 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:33.456573 30713 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.456981 30713 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P bf1e8f4327d44c4393b5dcacb70f1660: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:04:33.462544 30692 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.462932 30692 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P d5dcda017ad74e24b3c3c140b81a9c57: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:33.468045 30721 tablet_replica.cc:439] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } } }
I20260811 03:04:33.468382 30721 tablet_replica.cc:442] T bfdc59fba65a4a419793d686f3dacf3f P f49c98ba307b4295a7cb7bff38c44f2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:04:33.476039 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.466595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.27.109.3:38069)
0811 03:04:33.466964 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:33.466975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:33.467464 (+ 489us) server_negotiation.cc:407] Connection header received
0811 03:04:33.468239 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:33.468246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:33.468382 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:33.468574 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:33.469903 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.470952 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.472706 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.473479 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.473639 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.474295 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:33.475168 (+ 873us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:33.475346 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:33.475474 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 03:04:33.475574 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 03:04:33.494401 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.487618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44017 (local address 127.27.109.3:60155)
0811 03:04:33.488343 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:33.488360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:33.488490 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:33.488862 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:33.488865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:33.488874 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:04:33.489179 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.489188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.490848 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.490852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.491927 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.491940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.493079 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.493085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.493225 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.493292 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:04:33.493835 (+ 543us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:04:33.493846 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 03:04:33.493985 (+ 139us) client_negotiation.cc:240] Negotiation successful
0811 03:04:33.494100 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":386,"threads_started":1}
I20260811 03:04:33.494467 30681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.487673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:60155 (local address 127.27.109.2:44017)
0811 03:04:33.487915 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:33.487922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:33.488447 (+ 525us) server_negotiation.cc:407] Connection header received
0811 03:04:33.488626 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:33.488632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:33.488729 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:33.488830 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:33.489366 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.490703 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.492084 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.492857 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.493016 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.493537 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:33.493695 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:33.493811 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:33.494040 (+ 229us) server_negotiation.cc:299] Negotiation successful
0811 03:04:33.494133 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:04:33.545213 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.536065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34265 (local address 127.27.109.3:38069)
0811 03:04:33.536282 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:33.536289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:33.537076 (+ 787us) server_negotiation.cc:407] Connection header received
0811 03:04:33.537407 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:33.537414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:33.537526 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:33.537631 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:33.538537 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.539968 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.542491 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.543262 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.543363 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.544142 (+ 779us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:33.544382 (+ 240us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:33.544487 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:33.544700 (+ 213us) server_negotiation.cc:299] Negotiation successful
0811 03:04:33.544795 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 03:04:33.545432 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:33.536135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38069 (local address 127.27.109.2:34265)
0811 03:04:33.536962 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:33.536979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:33.537155 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:33.537828 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:33.537833 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:33.537853 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:04:33.538314 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.538336 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.540172 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.540178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.542278 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:33.542296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:33.543592 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:33.543601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:33.543832 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:33.543915 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:04:33.544514 (+ 599us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:04:33.544527 (+ 13us) client_negotiation.cc:769] Sending connection context
0811 03:04:33.544636 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 03:04:33.544813 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":435,"threads_started":1}
I20260811 03:04:33.599171 30096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:33.599501 30096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:33.604113 30096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: Sending DeleteTablet for 3 replicas of tablet dad2956c249c48cba85edeb9773a2861
I20260811 03:04:33.605506 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet dad2956c249c48cba85edeb9773a2861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:33 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:44814
I20260811 03:04:33.605613 30421 tablet_service.cc:1514] Processing DeleteTablet for tablet dad2956c249c48cba85edeb9773a2861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:33 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:54060
I20260811 03:04:33.605655 30580 tablet_service.cc:1514] Processing DeleteTablet for tablet dad2956c249c48cba85edeb9773a2861 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:33 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260811 03:04:33.608201 30096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:04:33.609187 30096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:33.618455 30255 tablet_service.cc:1467] Processing CreateTablet for tablet 2d57cb55bd9a44b8a820b98d28f4483d (DEFAULT_TABLE table=test-table [id=232b6db80dae4333bfe8224e77efe0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:33.619014 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d57cb55bd9a44b8a820b98d28f4483d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:33.619850 30746 tablet_replica.cc:330] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: stopping tablet replica
I20260811 03:04:33.620395 30746 raft_consensus.cc:2229] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:04:33.621162 30746 raft_consensus.cc:2258] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:33.623673 30420 tablet_service.cc:1467] Processing CreateTablet for tablet 2d57cb55bd9a44b8a820b98d28f4483d (DEFAULT_TABLE table=test-table [id=232b6db80dae4333bfe8224e77efe0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:33.624197 30420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d57cb55bd9a44b8a820b98d28f4483d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:33.627233 30745 tablet_replica.cc:330] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: stopping tablet replica
I20260811 03:04:33.627722 30745 raft_consensus.cc:2229] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:33.628119 30745 raft_consensus.cc:2258] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:33.630339 30746 ts_tablet_manager.cc:1905] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:33.634385 30580 tablet_service.cc:1467] Processing CreateTablet for tablet 2d57cb55bd9a44b8a820b98d28f4483d (DEFAULT_TABLE table=test-table [id=232b6db80dae4333bfe8224e77efe0d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:33.634655 30745 ts_tablet_manager.cc:1905] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:33.634904 30580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d57cb55bd9a44b8a820b98d28f4483d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:33.639228 30744 tablet_replica.cc:330] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: stopping tablet replica
I20260811 03:04:33.639761 30744 raft_consensus.cc:2229] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:33.640187 30744 raft_consensus.cc:2258] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:33.646509 30747 tablet_bootstrap.cc:492] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: Bootstrap starting.
I20260811 03:04:33.647977 30748 tablet_bootstrap.cc:492] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: Bootstrap starting.
I20260811 03:04:33.648617 30747 tablet_bootstrap.cc:654] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:33.651878 30748 tablet_bootstrap.cc:654] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:33.654191 30750 tablet_bootstrap.cc:492] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: Bootstrap starting.
I20260811 03:04:33.656980 30750 tablet_bootstrap.cc:654] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:33.664284 30745 ts_tablet_manager.cc:1918] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:33.664566 30745 log.cc:1196] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/dad2956c249c48cba85edeb9773a2861
I20260811 03:04:33.665278 30745 ts_tablet_manager.cc:1939] T dad2956c249c48cba85edeb9773a2861 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting consensus metadata
I20260811 03:04:33.666435 30747 tablet_bootstrap.cc:492] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: No bootstrap required, opened a new log
I20260811 03:04:33.666785 30747 ts_tablet_manager.cc:1397] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.003s
I20260811 03:04:33.667018 30744 ts_tablet_manager.cc:1905] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:33.667754 30747 raft_consensus.cc:348] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.668156 30747 raft_consensus.cc:374] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:33.668303 30747 raft_consensus.cc:729] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5dcda017ad74e24b3c3c140b81a9c57, State: Initialized, Role: FOLLOWER
I20260811 03:04:33.668685 30747 consensus_queue.cc:260] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.670224 30747 ts_tablet_manager.cc:1428] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260811 03:04:33.672592 30746 ts_tablet_manager.cc:1918] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:33.672809 30746 log.cc:1196] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/dad2956c249c48cba85edeb9773a2861
I20260811 03:04:33.673446 30746 ts_tablet_manager.cc:1939] T dad2956c249c48cba85edeb9773a2861 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting consensus metadata
I20260811 03:04:33.675441 30080 catalog_manager.cc:4841] TS f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069): tablet dad2956c249c48cba85edeb9773a2861 (table test-table [id=ffc49638ea4d47acab04c30cd062b052]) successfully deleted
I20260811 03:04:33.675989 30750 tablet_bootstrap.cc:492] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: No bootstrap required, opened a new log
I20260811 03:04:33.676154 30750 ts_tablet_manager.cc:1397] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20260811 03:04:33.676954 30750 raft_consensus.cc:348] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.677201 30750 raft_consensus.cc:374] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:33.677304 30750 raft_consensus.cc:729] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1e8f4327d44c4393b5dcacb70f1660, State: Initialized, Role: FOLLOWER
I20260811 03:04:33.677598 30750 consensus_queue.cc:260] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.678006 30744 ts_tablet_manager.cc:1918] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:33.678194 30744 log.cc:1196] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/dad2956c249c48cba85edeb9773a2861
I20260811 03:04:33.678825 30744 ts_tablet_manager.cc:1939] T dad2956c249c48cba85edeb9773a2861 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting consensus metadata
I20260811 03:04:33.680485 30750 ts_tablet_manager.cc:1428] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 03:04:33.680642 30081 catalog_manager.cc:4841] TS bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017): tablet dad2956c249c48cba85edeb9773a2861 (table test-table [id=ffc49638ea4d47acab04c30cd062b052]) successfully deleted
I20260811 03:04:33.684320 30748 tablet_bootstrap.cc:492] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: No bootstrap required, opened a new log
I20260811 03:04:33.685040 30080 catalog_manager.cc:4841] TS d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325): tablet dad2956c249c48cba85edeb9773a2861 (table test-table [id=ffc49638ea4d47acab04c30cd062b052]) successfully deleted
I20260811 03:04:33.685248 30748 ts_tablet_manager.cc:1397] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.013s
I20260811 03:04:33.686156 30748 raft_consensus.cc:348] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.686434 30748 raft_consensus.cc:374] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:33.686532 30748 raft_consensus.cc:729] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49c98ba307b4295a7cb7bff38c44f2b, State: Initialized, Role: FOLLOWER
I20260811 03:04:33.686805 30748 consensus_queue.cc:260] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.687134 30694 raft_consensus.cc:482] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:33.687481 30748 ts_tablet_manager.cc:1428] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 03:04:33.687328 30694 raft_consensus.cc:504] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.688138 30694 leader_election.cc:288] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325), bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017)
I20260811 03:04:33.689216 30281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d57cb55bd9a44b8a820b98d28f4483d" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" is_pre_election: true
I20260811 03:04:33.689514 30281 raft_consensus.cc:2454] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 0.
I20260811 03:04:33.689992 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d57cb55bd9a44b8a820b98d28f4483d" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" is_pre_election: true
I20260811 03:04:33.690294 30449 raft_consensus.cc:2454] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 0.
I20260811 03:04:33.690871 30523 leader_election.cc:302] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5dcda017ad74e24b3c3c140b81a9c57, f49c98ba307b4295a7cb7bff38c44f2b; no voters:
I20260811 03:04:33.691454 30721 raft_consensus.cc:2781] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:33.691571 30721 raft_consensus.cc:482] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:33.691717 30721 raft_consensus.cc:3037] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:33.695225 30721 raft_consensus.cc:504] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.696621 30721 leader_election.cc:288] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 election: Requested vote from peers d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325), bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017)
I20260811 03:04:33.696969 30281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d57cb55bd9a44b8a820b98d28f4483d" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5dcda017ad74e24b3c3c140b81a9c57"
I20260811 03:04:33.697224 30281 raft_consensus.cc:3037] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:33.697242 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d57cb55bd9a44b8a820b98d28f4483d" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1e8f4327d44c4393b5dcacb70f1660"
I20260811 03:04:33.697526 30449 raft_consensus.cc:3037] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:33.700388 30281 raft_consensus.cc:2454] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 1.
I20260811 03:04:33.700819 30523 leader_election.cc:302] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5dcda017ad74e24b3c3c140b81a9c57, f49c98ba307b4295a7cb7bff38c44f2b; no voters:
I20260811 03:04:33.701061 30449 raft_consensus.cc:2454] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 1.
I20260811 03:04:33.701164 30721 raft_consensus.cc:2781] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:33.701336 30721 raft_consensus.cc:686] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 1 LEADER]: Becoming Leader. State: Replica: f49c98ba307b4295a7cb7bff38c44f2b, State: Running, Role: LEADER
I20260811 03:04:33.701612 30721 consensus_queue.cc:237] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:33.729313 30096 catalog_manager.cc:5495] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3). New cstate: current_term: 1 leader_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } health_report { overall_health: HEALTHY } } }
I20260811 03:04:33.777361 30449 raft_consensus.cc:1261] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Refusing update from remote peer f49c98ba307b4295a7cb7bff38c44f2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:33.777984 30721 consensus_queue.cc:1035] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:33.780682 30281 raft_consensus.cc:1261] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Refusing update from remote peer f49c98ba307b4295a7cb7bff38c44f2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:33.781347 30721 consensus_queue.cc:1035] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:33.783550 30719 mvcc.cc:205] Tried to move back new op lower bound from 7317165972584820736 to 7317165972285014016. Current Snapshot: MvccSnapshot[applied={T|T < 7317165972584820736}]
I20260811 03:04:33.790578 30724 mvcc.cc:205] Tried to move back new op lower bound from 7317165972584820736 to 7317165972285014016. Current Snapshot: MvccSnapshot[applied={T|T < 7317165972584820736}]
W20260811 03:04:33.820892 30719 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 03:04:35.455345 30096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:35.455608 30096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:35.460693 30096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: Sending DeleteTablet for 3 replicas of tablet 2d57cb55bd9a44b8a820b98d28f4483d
I20260811 03:04:35.461710 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d57cb55bd9a44b8a820b98d28f4483d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:35 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:44814
I20260811 03:04:35.461794 30420 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d57cb55bd9a44b8a820b98d28f4483d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:35 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:54060
I20260811 03:04:35.462018 30580 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d57cb55bd9a44b8a820b98d28f4483d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:35 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260811 03:04:35.477269 30782 tablet_replica.cc:330] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: stopping tablet replica
I20260811 03:04:35.477450 30783 tablet_replica.cc:330] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: stopping tablet replica
I20260811 03:04:35.477583 30782 raft_consensus.cc:2229] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:35.477741 30783 raft_consensus.cc:2229] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:35.477911 30782 raft_consensus.cc:2258] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:35.478031 30783 raft_consensus.cc:2258] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:35.479151 30782 ts_tablet_manager.cc:1905] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:35.479216 30784 tablet_replica.cc:330] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: stopping tablet replica
I20260811 03:04:35.479353 30783 ts_tablet_manager.cc:1905] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:35.479637 30784 raft_consensus.cc:2229] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:04:35.480144 30784 raft_consensus.cc:2258] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:35.481172 30784 ts_tablet_manager.cc:1905] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:35.486756 30783 ts_tablet_manager.cc:1918] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 03:04:35.486935 30783 log.cc:1196] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/2d57cb55bd9a44b8a820b98d28f4483d
I20260811 03:04:35.487306 30782 ts_tablet_manager.cc:1918] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 03:04:35.487406 30783 ts_tablet_manager.cc:1939] T 2d57cb55bd9a44b8a820b98d28f4483d P bf1e8f4327d44c4393b5dcacb70f1660: Deleting consensus metadata
I20260811 03:04:35.487481 30782 log.cc:1196] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/2d57cb55bd9a44b8a820b98d28f4483d
I20260811 03:04:35.488001 30782 ts_tablet_manager.cc:1939] T 2d57cb55bd9a44b8a820b98d28f4483d P d5dcda017ad74e24b3c3c140b81a9c57: Deleting consensus metadata
I20260811 03:04:35.488837 30081 catalog_manager.cc:4841] TS bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017): tablet 2d57cb55bd9a44b8a820b98d28f4483d (table test-table [id=232b6db80dae4333bfe8224e77efe0d3]) successfully deleted
I20260811 03:04:35.489158 30784 ts_tablet_manager.cc:1918] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 03:04:35.489341 30784 log.cc:1196] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/2d57cb55bd9a44b8a820b98d28f4483d
I20260811 03:04:35.489276 30080 catalog_manager.cc:4841] TS d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325): tablet 2d57cb55bd9a44b8a820b98d28f4483d (table test-table [id=232b6db80dae4333bfe8224e77efe0d3]) successfully deleted
I20260811 03:04:35.489764 30784 ts_tablet_manager.cc:1939] T 2d57cb55bd9a44b8a820b98d28f4483d P f49c98ba307b4295a7cb7bff38c44f2b: Deleting consensus metadata
I20260811 03:04:35.490962 30080 catalog_manager.cc:4841] TS f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069): tablet 2d57cb55bd9a44b8a820b98d28f4483d (table test-table [id=232b6db80dae4333bfe8224e77efe0d3]) successfully deleted
Aug 11 03:04:35 dist-test-slave-8tdl krb5kdc[30046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 03:04:35 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:35.532145 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:35.515991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.27.109.62:33375)
0811 03:04:35.516701 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:35.516708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:35.516735 (+ 27us) server_negotiation.cc:407] Connection header received
0811 03:04:35.516814 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:35.516818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:35.516879 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:35.517016 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:35.518411 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:35.519340 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:35.520670 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:35.521134 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:35.521277 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:35.526142 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:35.526147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:35.526150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:35.526190 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:35.529167 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:35.529924 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:35.529930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:35.529934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:35.530088 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:35.530523 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:35.530527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:35.530529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:35.530907 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:35.531172 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:35.531489 (+ 317us) server_negotiation.cc:299] Negotiation successful
0811 03:04:35.531740 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":354,"threads_started":1}
I20260811 03:04:35.536199 30096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:04:35.537180 30096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:35.544797 30255 tablet_service.cc:1467] Processing CreateTablet for tablet f2758cd0857b4de3b5f0382790abf948 (DEFAULT_TABLE table=test-table [id=c64d800ff78c486fbc4b5c18a4c6af17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:35.545154 30420 tablet_service.cc:1467] Processing CreateTablet for tablet f2758cd0857b4de3b5f0382790abf948 (DEFAULT_TABLE table=test-table [id=c64d800ff78c486fbc4b5c18a4c6af17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:35.545183 30580 tablet_service.cc:1467] Processing CreateTablet for tablet f2758cd0857b4de3b5f0382790abf948 (DEFAULT_TABLE table=test-table [id=c64d800ff78c486fbc4b5c18a4c6af17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:35.545392 30255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2758cd0857b4de3b5f0382790abf948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:35.545634 30580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2758cd0857b4de3b5f0382790abf948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:35.545662 30420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2758cd0857b4de3b5f0382790abf948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:35.552037 30796 tablet_bootstrap.cc:492] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: Bootstrap starting.
I20260811 03:04:35.552937 30797 tablet_bootstrap.cc:492] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: Bootstrap starting.
I20260811 03:04:35.553400 30798 tablet_bootstrap.cc:492] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: Bootstrap starting.
I20260811 03:04:35.554934 30796 tablet_bootstrap.cc:654] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:35.554925 30797 tablet_bootstrap.cc:654] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:35.555291 30798 tablet_bootstrap.cc:654] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:35.557569 30796 tablet_bootstrap.cc:492] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: No bootstrap required, opened a new log
I20260811 03:04:35.557569 30797 tablet_bootstrap.cc:492] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: No bootstrap required, opened a new log
I20260811 03:04:35.557735 30797 ts_tablet_manager.cc:1397] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260811 03:04:35.557735 30796 ts_tablet_manager.cc:1397] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260811 03:04:35.558645 30796 raft_consensus.cc:348] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.558651 30797 raft_consensus.cc:348] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.558945 30797 raft_consensus.cc:374] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:35.558945 30796 raft_consensus.cc:374] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:35.559043 30797 raft_consensus.cc:729] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1e8f4327d44c4393b5dcacb70f1660, State: Initialized, Role: FOLLOWER
I20260811 03:04:35.559043 30796 raft_consensus.cc:729] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5dcda017ad74e24b3c3c140b81a9c57, State: Initialized, Role: FOLLOWER
I20260811 03:04:35.559286 30796 consensus_queue.cc:260] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.559363 30797 consensus_queue.cc:260] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.559970 30797 ts_tablet_manager.cc:1428] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 03:04:35.563174 30796 ts_tablet_manager.cc:1428] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260811 03:04:35.564621 30798 tablet_bootstrap.cc:492] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: No bootstrap required, opened a new log
I20260811 03:04:35.564795 30798 ts_tablet_manager.cc:1397] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260811 03:04:35.565659 30798 raft_consensus.cc:348] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.565932 30798 raft_consensus.cc:374] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:35.566037 30798 raft_consensus.cc:729] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49c98ba307b4295a7cb7bff38c44f2b, State: Initialized, Role: FOLLOWER
I20260811 03:04:35.566294 30798 consensus_queue.cc:260] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.566874 30798 ts_tablet_manager.cc:1428] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260811 03:04:35.887248 30773 raft_consensus.cc:482] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:35.887593 30773 raft_consensus.cc:504] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.888718 30773 leader_election.cc:288] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325), bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017)
I20260811 03:04:35.889441 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2758cd0857b4de3b5f0382790abf948" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" is_pre_election: true
I20260811 03:04:35.889418 30281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2758cd0857b4de3b5f0382790abf948" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" is_pre_election: true
I20260811 03:04:35.889737 30449 raft_consensus.cc:2454] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 0.
I20260811 03:04:35.889756 30281 raft_consensus.cc:2454] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 0.
I20260811 03:04:35.890292 30524 leader_election.cc:302] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf1e8f4327d44c4393b5dcacb70f1660, f49c98ba307b4295a7cb7bff38c44f2b; no voters:
I20260811 03:04:35.890590 30773 raft_consensus.cc:2781] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:35.890770 30773 raft_consensus.cc:482] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:35.890988 30773 raft_consensus.cc:3037] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:35.893520 30773 raft_consensus.cc:504] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.894304 30773 leader_election.cc:288] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 election: Requested vote from peers d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325), bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017)
I20260811 03:04:35.894834 30281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2758cd0857b4de3b5f0382790abf948" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5dcda017ad74e24b3c3c140b81a9c57"
I20260811 03:04:35.894958 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2758cd0857b4de3b5f0382790abf948" candidate_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1e8f4327d44c4393b5dcacb70f1660"
I20260811 03:04:35.895162 30281 raft_consensus.cc:3037] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:35.895275 30449 raft_consensus.cc:3037] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:35.897480 30281 raft_consensus.cc:2454] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 1.
I20260811 03:04:35.897872 30449 raft_consensus.cc:2454] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49c98ba307b4295a7cb7bff38c44f2b in term 1.
I20260811 03:04:35.898089 30523 leader_election.cc:302] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5dcda017ad74e24b3c3c140b81a9c57, f49c98ba307b4295a7cb7bff38c44f2b; no voters:
I20260811 03:04:35.898452 30773 raft_consensus.cc:2781] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:35.898648 30773 raft_consensus.cc:686] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 LEADER]: Becoming Leader. State: Replica: f49c98ba307b4295a7cb7bff38c44f2b, State: Running, Role: LEADER
I20260811 03:04:35.899012 30773 consensus_queue.cc:237] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:35.902176 30095 catalog_manager.cc:5495] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3). New cstate: current_term: 1 leader_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } health_report { overall_health: HEALTHY } } }
I20260811 03:04:35.935978 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:35.930869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.27.109.3:38069)
0811 03:04:35.931368 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:35.931372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:35.931398 (+ 26us) server_negotiation.cc:407] Connection header received
0811 03:04:35.931466 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:35.931470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:35.931531 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:35.931616 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:35.931960 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:35.932771 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:35.934059 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:35.934472 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:35.934566 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:35.935128 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:35.935232 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:35.935300 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:35.935471 (+ 171us) server_negotiation.cc:299] Negotiation successful
0811 03:04:35.935575 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":253,"threads_started":1}
I20260811 03:04:35.945987 30449 raft_consensus.cc:1261] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Refusing update from remote peer f49c98ba307b4295a7cb7bff38c44f2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:35.946171 30281 raft_consensus.cc:1261] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Refusing update from remote peer f49c98ba307b4295a7cb7bff38c44f2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:35.946704 30773 consensus_queue.cc:1035] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:35.947108 30696 consensus_queue.cc:1035] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:35.952658 30761 mvcc.cc:205] Tried to move back new op lower bound from 7317165981469609984 to 7317165981284896768. Current Snapshot: MvccSnapshot[applied={T|T < 7317165981469609984}]
I20260811 03:04:36.010011 30095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:36.010306 30095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:36.013878 30095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: Sending DeleteTablet for 3 replicas of tablet f2758cd0857b4de3b5f0382790abf948
I20260811 03:04:36.014946 30255 tablet_service.cc:1514] Processing DeleteTablet for tablet f2758cd0857b4de3b5f0382790abf948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:36 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:44814
I20260811 03:04:36.015105 30420 tablet_service.cc:1514] Processing DeleteTablet for tablet f2758cd0857b4de3b5f0382790abf948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:36 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:54060
I20260811 03:04:36.015264 30580 tablet_service.cc:1514] Processing DeleteTablet for tablet f2758cd0857b4de3b5f0382790abf948 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:36 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260811 03:04:36.017408 30095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:04:36.018741 30095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:36.027688 30256 tablet_service.cc:1467] Processing CreateTablet for tablet b0813f4633f24a4bbceb378685bea0a8 (DEFAULT_TABLE table=test-table [id=6da1436fb8134aebbb0f174039b055e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:36.028009 30421 tablet_service.cc:1467] Processing CreateTablet for tablet b0813f4633f24a4bbceb378685bea0a8 (DEFAULT_TABLE table=test-table [id=6da1436fb8134aebbb0f174039b055e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:36.028353 30256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0813f4633f24a4bbceb378685bea0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:36.028455 30421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0813f4633f24a4bbceb378685bea0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:36.030195 30820 tablet_replica.cc:330] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: stopping tablet replica
I20260811 03:04:36.030428 30820 raft_consensus.cc:2229] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:36.030664 30820 raft_consensus.cc:2258] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:36.031744 30579 tablet_service.cc:1467] Processing CreateTablet for tablet b0813f4633f24a4bbceb378685bea0a8 (DEFAULT_TABLE table=test-table [id=6da1436fb8134aebbb0f174039b055e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:36.031965 30820 ts_tablet_manager.cc:1905] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:36.032217 30579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0813f4633f24a4bbceb378685bea0a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:36.033771 30822 tablet_replica.cc:330] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: stopping tablet replica
I20260811 03:04:36.034031 30822 raft_consensus.cc:2229] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:04:36.034561 30822 raft_consensus.cc:2258] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:36.037447 30821 tablet_replica.cc:330] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: stopping tablet replica
I20260811 03:04:36.037679 30821 raft_consensus.cc:2229] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:36.037931 30821 raft_consensus.cc:2258] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:36.039085 30797 tablet_bootstrap.cc:492] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: Bootstrap starting.
I20260811 03:04:36.039178 30820 ts_tablet_manager.cc:1918] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:36.039316 30820 log.cc:1196] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/f2758cd0857b4de3b5f0382790abf948
I20260811 03:04:36.040236 30820 ts_tablet_manager.cc:1939] T f2758cd0857b4de3b5f0382790abf948 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting consensus metadata
I20260811 03:04:36.040563 30821 ts_tablet_manager.cc:1905] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:36.041126 30797 tablet_bootstrap.cc:654] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:36.042331 30080 catalog_manager.cc:4841] TS d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325): tablet f2758cd0857b4de3b5f0382790abf948 (table test-table [id=c64d800ff78c486fbc4b5c18a4c6af17]) successfully deleted
I20260811 03:04:36.045293 30798 tablet_bootstrap.cc:492] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: Bootstrap starting.
I20260811 03:04:36.046471 30822 ts_tablet_manager.cc:1905] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:36.046792 30796 tablet_bootstrap.cc:492] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: Bootstrap starting.
I20260811 03:04:36.047472 30797 tablet_bootstrap.cc:492] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: No bootstrap required, opened a new log
I20260811 03:04:36.047127 30798 tablet_bootstrap.cc:654] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:36.049031 30796 tablet_bootstrap.cc:654] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:36.049841 30797 ts_tablet_manager.cc:1397] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260811 03:04:36.050655 30797 raft_consensus.cc:348] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.050930 30797 raft_consensus.cc:374] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:36.051021 30797 raft_consensus.cc:729] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1e8f4327d44c4393b5dcacb70f1660, State: Initialized, Role: FOLLOWER
I20260811 03:04:36.051363 30797 consensus_queue.cc:260] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.053061 30797 ts_tablet_manager.cc:1428] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260811 03:04:36.056428 30821 ts_tablet_manager.cc:1918] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:36.056609 30821 log.cc:1196] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/f2758cd0857b4de3b5f0382790abf948
I20260811 03:04:36.056628 30822 ts_tablet_manager.cc:1918] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:04:36.056798 30822 log.cc:1196] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/f2758cd0857b4de3b5f0382790abf948
I20260811 03:04:36.057140 30821 ts_tablet_manager.cc:1939] T f2758cd0857b4de3b5f0382790abf948 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting consensus metadata
I20260811 03:04:36.057482 30822 ts_tablet_manager.cc:1939] T f2758cd0857b4de3b5f0382790abf948 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting consensus metadata
I20260811 03:04:36.058714 30081 catalog_manager.cc:4841] TS bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017): tablet f2758cd0857b4de3b5f0382790abf948 (table test-table [id=c64d800ff78c486fbc4b5c18a4c6af17]) successfully deleted
I20260811 03:04:36.058800 30798 tablet_bootstrap.cc:492] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: No bootstrap required, opened a new log
I20260811 03:04:36.058974 30798 ts_tablet_manager.cc:1397] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260811 03:04:36.059257 30080 catalog_manager.cc:4841] TS f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069): tablet f2758cd0857b4de3b5f0382790abf948 (table test-table [id=c64d800ff78c486fbc4b5c18a4c6af17]) successfully deleted
I20260811 03:04:36.059841 30798 raft_consensus.cc:348] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.060091 30798 raft_consensus.cc:374] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:36.060181 30798 raft_consensus.cc:729] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49c98ba307b4295a7cb7bff38c44f2b, State: Initialized, Role: FOLLOWER
I20260811 03:04:36.060468 30798 consensus_queue.cc:260] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.061053 30798 ts_tablet_manager.cc:1428] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 03:04:36.061097 30796 tablet_bootstrap.cc:492] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: No bootstrap required, opened a new log
I20260811 03:04:36.061298 30796 ts_tablet_manager.cc:1397] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260811 03:04:36.062054 30796 raft_consensus.cc:348] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.062294 30796 raft_consensus.cc:374] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:36.062392 30796 raft_consensus.cc:729] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5dcda017ad74e24b3c3c140b81a9c57, State: Initialized, Role: FOLLOWER
I20260811 03:04:36.062662 30796 consensus_queue.cc:260] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.063295 30796 ts_tablet_manager.cc:1428] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 03:04:36.071041 30805 raft_consensus.cc:482] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:36.071339 30805 raft_consensus.cc:504] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.072988 30805 leader_election.cc:288] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017), f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069)
I20260811 03:04:36.080374 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.073869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:40913 (local address 127.27.109.3:38069)
0811 03:04:36.074110 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:36.074116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:36.074387 (+ 271us) server_negotiation.cc:407] Connection header received
0811 03:04:36.074610 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:36.074619 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:36.074758 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:36.074906 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:36.075698 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.076448 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.078178 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.079285 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.079450 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.079525 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:36.079953 (+ 428us) server_negotiation.cc:299] Negotiation successful
0811 03:04:36.080097 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:04:36.080451 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.073271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38069 (local address 127.27.109.1:40913)
0811 03:04:36.074273 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:36.074296 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:36.074448 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:36.075114 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:36.075119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:36.075142 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:36.075521 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:36.075530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.076585 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.076592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:36.078026 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:36.078038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.079503 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.079508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:36.079680 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.079727 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 03:04:36.079895 (+ 168us) client_negotiation.cc:240] Negotiation successful
0811 03:04:36.080058 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":726,"threads_started":1}
I20260811 03:04:36.081161 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.073370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44017 (local address 127.27.109.1:55899)
0811 03:04:36.075333 (+ 1963us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:36.075350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:36.075527 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:36.075871 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:36.075875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:36.075900 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:36.076233 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:36.076242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.077298 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.077304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:36.078669 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:36.078732 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.080323 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.080329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:36.080511 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.080558 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 03:04:36.080656 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 03:04:36.080787 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1838,"thread_start_us":1512,"threads_started":1}
I20260811 03:04:36.081199 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.073331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:55899 (local address 127.27.109.2:44017)
0811 03:04:36.074056 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:36.074062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:36.075450 (+ 1388us) server_negotiation.cc:407] Connection header received
0811 03:04:36.075679 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:36.075683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:36.075745 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:36.075853 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:36.076412 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.077149 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.078904 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.080092 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.080262 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.080326 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:36.080724 (+ 398us) server_negotiation.cc:299] Negotiation successful
0811 03:04:36.080870 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":390,"threads_started":1}
I20260811 03:04:36.081130 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0813f4633f24a4bbceb378685bea0a8" candidate_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" is_pre_election: true
I20260811 03:04:36.081413 30608 raft_consensus.cc:2454] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5dcda017ad74e24b3c3c140b81a9c57 in term 0.
I20260811 03:04:36.081866 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0813f4633f24a4bbceb378685bea0a8" candidate_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" is_pre_election: true
I20260811 03:04:36.081961 30198 leader_election.cc:302] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5dcda017ad74e24b3c3c140b81a9c57, f49c98ba307b4295a7cb7bff38c44f2b; no voters:
I20260811 03:04:36.082181 30449 raft_consensus.cc:2454] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5dcda017ad74e24b3c3c140b81a9c57 in term 0.
I20260811 03:04:36.082444 30805 raft_consensus.cc:2781] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:36.082590 30805 raft_consensus.cc:482] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:36.082726 30805 raft_consensus.cc:3037] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:36.084911 30805 raft_consensus.cc:504] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.085489 30805 leader_election.cc:288] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [CANDIDATE]: Term 1 election: Requested vote from peers bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017), f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069)
I20260811 03:04:36.086066 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0813f4633f24a4bbceb378685bea0a8" candidate_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49c98ba307b4295a7cb7bff38c44f2b"
I20260811 03:04:36.086071 30449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0813f4633f24a4bbceb378685bea0a8" candidate_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1e8f4327d44c4393b5dcacb70f1660"
I20260811 03:04:36.086283 30608 raft_consensus.cc:3037] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:36.086314 30449 raft_consensus.cc:3037] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:36.088356 30608 raft_consensus.cc:2454] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5dcda017ad74e24b3c3c140b81a9c57 in term 1.
I20260811 03:04:36.088553 30449 raft_consensus.cc:2454] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5dcda017ad74e24b3c3c140b81a9c57 in term 1.
I20260811 03:04:36.088838 30198 leader_election.cc:302] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5dcda017ad74e24b3c3c140b81a9c57, f49c98ba307b4295a7cb7bff38c44f2b; no voters:
I20260811 03:04:36.089152 30805 raft_consensus.cc:2781] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:36.090030 30805 raft_consensus.cc:686] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 LEADER]: Becoming Leader. State: Replica: d5dcda017ad74e24b3c3c140b81a9c57, State: Running, Role: LEADER
I20260811 03:04:36.090435 30805 consensus_queue.cc:237] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } }
I20260811 03:04:36.094270 30095 catalog_manager.cc:5495] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5dcda017ad74e24b3c3c140b81a9c57" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 } health_report { overall_health: UNKNOWN } } }
I20260811 03:04:36.122922 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.116847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.27.109.1:41325)
0811 03:04:36.117481 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:36.117486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:36.117505 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:04:36.117601 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:36.117604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:36.117682 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:36.117894 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:36.118292 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.119192 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.120508 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.120959 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.121064 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.121500 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:36.122148 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:36.122259 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:36.122401 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 03:04:36.122470 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":371,"threads_started":1}
I20260811 03:04:36.139623 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.133972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:47771 (local address 127.27.109.2:44017)
0811 03:04:36.134166 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:36.134170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:36.134771 (+ 601us) server_negotiation.cc:407] Connection header received
0811 03:04:36.135037 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:36.135042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:36.135115 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:36.135200 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:36.135743 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.136527 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.137866 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.138350 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.138449 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.139004 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:36.139111 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:36.139183 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:36.139316 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 03:04:36.139382 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 03:04:36.139696 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.133832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44017 (local address 127.27.109.1:47771)
0811 03:04:36.134640 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:36.134662 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:36.134812 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:36.135230 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:36.135233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:36.135243 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:04:36.135578 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:36.135587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.136660 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.136664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:36.137732 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:36.137744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.138509 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.138514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:36.138725 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.138784 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:04:36.139201 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:04:36.139217 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 03:04:36.139279 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 03:04:36.139384 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":440,"threads_started":1}
I20260811 03:04:36.148813 30608 raft_consensus.cc:1261] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Refusing update from remote peer d5dcda017ad74e24b3c3c140b81a9c57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:36.148923 30449 raft_consensus.cc:1261] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Refusing update from remote peer d5dcda017ad74e24b3c3c140b81a9c57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:36.149549 30829 consensus_queue.cc:1035] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49c98ba307b4295a7cb7bff38c44f2b" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:36.149945 30805 consensus_queue.cc:1035] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1e8f4327d44c4393b5dcacb70f1660" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:36.182518 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.176332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:36739 (local address 127.27.109.1:41325)
0811 03:04:36.176550 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:36.176556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:36.177191 (+ 635us) server_negotiation.cc:407] Connection header received
0811 03:04:36.177375 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:36.177383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:36.177468 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:36.177550 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:36.178173 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.179266 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.180800 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.181210 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.181308 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.181764 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:36.181867 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:36.181933 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:36.182140 (+ 207us) server_negotiation.cc:299] Negotiation successful
0811 03:04:36.182235 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:04:36.182667 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:36.176254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41325 (local address 127.27.109.2:36739)
0811 03:04:36.177071 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:36.177092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:36.177237 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:36.177629 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:36.177634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:36.177648 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:04:36.177962 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:36.177980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.179463 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.179474 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:36.180661 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:36.180675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:36.181380 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:36.181411 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:36.181556 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:36.181597 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:04:36.181980 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:04:36.181996 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 03:04:36.182084 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 03:04:36.182222 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":389,"threads_started":1}
W20260811 03:04:36.184191 30761 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 03:04:37.808616 30095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:37.808952 30095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:37.814250 30095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a856997b14c4293952cc8dc2a620c52: Sending DeleteTablet for 3 replicas of tablet b0813f4633f24a4bbceb378685bea0a8
I20260811 03:04:37.815526 30256 tablet_service.cc:1514] Processing DeleteTablet for tablet b0813f4633f24a4bbceb378685bea0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:37 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:44814
I20260811 03:04:37.815517 30420 tablet_service.cc:1514] Processing DeleteTablet for tablet b0813f4633f24a4bbceb378685bea0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:37 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:54060
I20260811 03:04:37.815554 30579 tablet_service.cc:1514] Processing DeleteTablet for tablet b0813f4633f24a4bbceb378685bea0a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:37 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57296
I20260811 03:04:37.816154 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30181
I20260811 03:04:37.830974 30868 tablet_replica.cc:330] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: stopping tablet replica
I20260811 03:04:37.830978 30869 tablet_replica.cc:330] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: stopping tablet replica
I20260811 03:04:37.831247 30868 raft_consensus.cc:2229] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:04:37.831236 30869 raft_consensus.cc:2229] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:37.831564 30869 raft_consensus.cc:2258] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:37.832401 30868 raft_consensus.cc:2258] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:37.832430 30870 tablet_replica.cc:330] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: stopping tablet replica
I20260811 03:04:37.832683 30870 raft_consensus.cc:2229] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:37.832726 30869 ts_tablet_manager.cc:1905] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:37.832959 30870 raft_consensus.cc:2258] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:37.833384 30868 ts_tablet_manager.cc:1905] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:37.834437 30870 ts_tablet_manager.cc:1905] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:37.839880 30869 ts_tablet_manager.cc:1918] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 03:04:37.840065 30869 log.cc:1196] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/b0813f4633f24a4bbceb378685bea0a8
I20260811 03:04:37.840302 30868 ts_tablet_manager.cc:1918] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 03:04:37.840456 30868 log.cc:1196] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/b0813f4633f24a4bbceb378685bea0a8
I20260811 03:04:37.840626 30869 ts_tablet_manager.cc:1939] T b0813f4633f24a4bbceb378685bea0a8 P bf1e8f4327d44c4393b5dcacb70f1660: Deleting consensus metadata
I20260811 03:04:37.840931 30868 ts_tablet_manager.cc:1939] T b0813f4633f24a4bbceb378685bea0a8 P d5dcda017ad74e24b3c3c140b81a9c57: Deleting consensus metadata
I20260811 03:04:37.842192 30081 catalog_manager.cc:4841] TS bf1e8f4327d44c4393b5dcacb70f1660 (127.27.109.2:44017): tablet b0813f4633f24a4bbceb378685bea0a8 (table test-table [id=6da1436fb8134aebbb0f174039b055e9]) successfully deleted
I20260811 03:04:37.842234 30870 ts_tablet_manager.cc:1918] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 03:04:37.842154 30080 catalog_manager.cc:4841] TS d5dcda017ad74e24b3c3c140b81a9c57 (127.27.109.1:41325): tablet b0813f4633f24a4bbceb378685bea0a8 (table test-table [id=6da1436fb8134aebbb0f174039b055e9]) successfully deleted
I20260811 03:04:37.842383 30870 log.cc:1196] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/b0813f4633f24a4bbceb378685bea0a8
I20260811 03:04:37.842998 30870 ts_tablet_manager.cc:1939] T b0813f4633f24a4bbceb378685bea0a8 P f49c98ba307b4295a7cb7bff38c44f2b: Deleting consensus metadata
I20260811 03:04:37.844398 30080 catalog_manager.cc:4841] TS f49c98ba307b4295a7cb7bff38c44f2b (127.27.109.3:38069): tablet b0813f4633f24a4bbceb378685bea0a8 (table test-table [id=6da1436fb8134aebbb0f174039b055e9]) successfully deleted
Aug 11 03:04:37 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:37.853309 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:37.818065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.27.109.1:41325)
0811 03:04:37.831260 (+ 13195us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:37.831271 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:37.831312 (+ 41us) server_negotiation.cc:407] Connection header received
0811 03:04:37.831398 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:37.831407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:37.831593 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:37.831743 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:37.835673 (+ 3930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:37.836961 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:37.839098 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:37.839858 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:37.840084 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:37.846421 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:37.846445 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:37.846481 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:37.846542 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:37.849814 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:37.850602 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:37.850610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:37.850613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:37.850916 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:37.851419 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:37.851426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:37.851429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:37.851930 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:37.852278 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:37.852645 (+ 367us) server_negotiation.cc:299] Negotiation successful
0811 03:04:37.852845 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13014,"thread_start_us":13794,"threads_started":1}
W20260811 03:04:37.854136 30331 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49806
W20260811 03:04:37.854936 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30181: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:37.855208 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 30181
I20260811 03:04:37.887019 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30343
Aug 11 03:04:37 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:04:37.904778 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:37.887892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.27.109.2:44017)
0811 03:04:37.888541 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:37.888547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:37.888565 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:04:37.888627 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:37.888631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:37.888694 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:37.888797 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:37.890178 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:37.891200 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:37.892562 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:37.893030 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:37.893138 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:37.898084 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:37.898100 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:37.898117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:37.898163 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:37.901264 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:37.902285 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:37.902297 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:37.902300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:37.902509 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:37.903076 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:37.903080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:37.903082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:37.903542 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:37.903817 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:37.904141 (+ 324us) server_negotiation.cc:299] Negotiation successful
0811 03:04:37.904342 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":319,"threads_started":1}
W20260811 03:04:37.905623 30493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52632
W20260811 03:04:37.906386 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30343: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:37.906589 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 30343
I20260811 03:04:37.938616 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30505
Aug 11 03:04:37 dist-test-slave-8tdl krb5kdc[30046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417475, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:04:37.956970 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:37.939520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.27.109.3:38069)
0811 03:04:37.940056 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:37.940063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:37.940079 (+ 16us) server_negotiation.cc:407] Connection header received
0811 03:04:37.940137 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:37.940141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:37.940202 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:37.940297 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:37.941629 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:37.942443 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:37.944164 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:37.944637 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:37.944762 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:37.949923 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:37.949943 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:37.949960 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:37.950009 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:37.953535 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:37.954332 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:37.954340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:37.954342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:37.954495 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:37.955181 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:37.955185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:37.955187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:37.955619 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:37.955885 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:37.956323 (+ 438us) server_negotiation.cc:299] Negotiation successful
0811 03:04:37.956526 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
W20260811 03:04:37.957746 30654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41680
W20260811 03:04:37.958451 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30505: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:37.958638 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 30505
I20260811 03:04:37.987505 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30059
I20260811 03:04:37.988858 30145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:38.111254 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 30059
2026-08-11T03:04:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9339 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:38 dist-test-slave-8tdl krb5kdc[30878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:38 dist-test-slave-8tdl krb5kdc[30878](info): set up 2 sockets
Aug 11 03:04:38 dist-test-slave-8tdl krb5kdc[30878](info): commencing operation
krb5kdc: starting...
W20260811 03:04:39.317631 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:39 dist-test-slave-8tdl krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:04:39.562569 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:44945
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:44945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:39.958082 30891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:39.958505 30891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:39.958712 30891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:39.968143 30891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:39.968375 30891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:39.968510 30891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:39.968648 30891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:39.968743 30891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:39.981357 30891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:44945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:44945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:39.983254 30891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:39.985029 30891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:40.001176 30896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:40.001101 30897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:40.001144 30899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:40.002157 30891 server_base.cc:1034] running on GCE node
I20260811 03:04:40.003212 30891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:40.005169 30891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:40.006758 30891 hybrid_clock.cc:648] HybridClock initialized: now 1786417480006592 us; error 84 us; skew 500 ppm
I20260811 03:04:40.007292 30891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:40 dist-test-slave-8tdl krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:40.011696 30891 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:40.014165 30891 webserver.cc:460] Webserver started at http://127.27.109.62:46813/ using document root <none> and password file <none>
I20260811 03:04:40.015323 30891 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:40.015501 30891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:40.015834 30891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:40.019826 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "2474979fbe4c4ee8b0e125131d76cc7e"
format_stamp: "Formatted at 2026-08-11 03:04:40 on dist-test-slave-8tdl"
server_key: "88c636c33f4f302eaf7772808c475ede"
server_key_iv: "a5494a40e8d2e7373a7714c4735dfd5e"
server_key_version: "encryptionkey@0"
I20260811 03:04:40.020816 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "2474979fbe4c4ee8b0e125131d76cc7e"
format_stamp: "Formatted at 2026-08-11 03:04:40 on dist-test-slave-8tdl"
server_key: "88c636c33f4f302eaf7772808c475ede"
server_key_iv: "a5494a40e8d2e7373a7714c4735dfd5e"
server_key_version: "encryptionkey@0"
I20260811 03:04:40.027815 30891 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260811 03:04:40.032656 30906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:40.034706 30891 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260811 03:04:40.035014 30891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "2474979fbe4c4ee8b0e125131d76cc7e"
format_stamp: "Formatted at 2026-08-11 03:04:40 on dist-test-slave-8tdl"
server_key: "88c636c33f4f302eaf7772808c475ede"
server_key_iv: "a5494a40e8d2e7373a7714c4735dfd5e"
server_key_version: "encryptionkey@0"
I20260811 03:04:40.035282 30891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:40.055009 30891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:40.060827 30891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:40.061200 30891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:40.084654 30891 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:44945
I20260811 03:04:40.084713 30968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:44945 every 8 connection(s)
I20260811 03:04:40.086617 30891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:40.090870 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 30891
I20260811 03:04:40.091163 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:40.092070 28084 external_mini_cluster.cc:1428] Setting key a2ec1ce915651a04855d58aaa66d74f4
I20260811 03:04:40.092773 30969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:40.110862 30969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e: Bootstrap starting.
Aug 11 03:04:40 dist-test-slave-8tdl krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:40.115641 30969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:40.117125 30969 log.cc:824] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:40.120911 30969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e: No bootstrap required, opened a new log
I20260811 03:04:40.122794 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:40.095060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.27.109.62:44945)
0811 03:04:40.096092 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:40.096107 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:40.096165 (+ 58us) server_negotiation.cc:407] Connection header received
0811 03:04:40.098466 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:40.098543 (+ 77us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:40.099235 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:40.099708 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:40.102060 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:40.103694 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:40.105809 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:40.107027 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:40.107248 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:40.113179 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:40.113189 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:40.113208 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:40.113270 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:40.117442 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:40.118473 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:40.118483 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:40.118492 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:40.118818 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:40.119383 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:40.119393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:40.119395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:40.120046 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:40.120426 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:40.120999 (+ 573us) server_negotiation.cc:299] Negotiation successful
0811 03:04:40.121540 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":459,"threads_started":1}
I20260811 03:04:40.131050 30969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2474979fbe4c4ee8b0e125131d76cc7e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44945 } }
I20260811 03:04:40.131598 30969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:40.131738 30969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2474979fbe4c4ee8b0e125131d76cc7e, State: Initialized, Role: FOLLOWER
I20260811 03:04:40.132433 30969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2474979fbe4c4ee8b0e125131d76cc7e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44945 } }
I20260811 03:04:40.132750 30969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:40.132925 30969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:40.133136 30969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:40.135870 30969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2474979fbe4c4ee8b0e125131d76cc7e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44945 } }
I20260811 03:04:40.136579 30969 leader_election.cc:302] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2474979fbe4c4ee8b0e125131d76cc7e; no voters:
I20260811 03:04:40.137315 30969 leader_election.cc:288] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:40.137535 30976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:40.139055 30976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [term 1 LEADER]: Becoming Leader. State: Replica: 2474979fbe4c4ee8b0e125131d76cc7e, State: Running, Role: LEADER
I20260811 03:04:40.139784 30976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2474979fbe4c4ee8b0e125131d76cc7e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44945 } }
I20260811 03:04:40.140051 30969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:40.144008 30977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2474979fbe4c4ee8b0e125131d76cc7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2474979fbe4c4ee8b0e125131d76cc7e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44945 } } }
I20260811 03:04:40.144654 30977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:40.143980 30978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2474979fbe4c4ee8b0e125131d76cc7e. Latest consensus state: current_term: 1 leader_uuid: "2474979fbe4c4ee8b0e125131d76cc7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2474979fbe4c4ee8b0e125131d76cc7e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44945 } } }
I20260811 03:04:40.145364 30978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:40.157423 30986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:40.164328 30986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:40.176445 30986 catalog_manager.cc:1347] Generated new cluster ID: 20ebde2f7065415ba93344c9bf148bec
I20260811 03:04:40.176637 30986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:40.203251 30986 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:40.204941 30986 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:40.216713 30986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2474979fbe4c4ee8b0e125131d76cc7e: Generated new TSK 0
I20260811 03:04:40.217861 30986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:04:40.287773 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44945
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:40.678793 30999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:40.679138 30999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:40.679327 30999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:40.686911 30999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:40.687165 30999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:40.687354 30999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:40.697495 30999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:40.698976 30999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:40.700410 30999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:40.714993 31004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:40.714967 31005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:40.714922 31007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:40.716110 30999 server_base.cc:1034] running on GCE node
I20260811 03:04:40.716811 30999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:40.717912 30999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:40.719161 30999 hybrid_clock.cc:648] HybridClock initialized: now 1786417480719105 us; error 47 us; skew 500 ppm
I20260811 03:04:40.719604 30999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:40 dist-test-slave-8tdl krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:40.723172 30999 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:04:40.724783 30999 webserver.cc:460] Webserver started at http://127.27.109.1:34593/ using document root <none> and password file <none>
I20260811 03:04:40.725481 30999 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:40.725589 30999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:40.725852 30999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:40.730321 30999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "64e451830adc4cccabb39a37894d6ac0"
format_stamp: "Formatted at 2026-08-11 03:04:40 on dist-test-slave-8tdl"
server_key: "6859080740b25672ca3f245658f4074f"
server_key_iv: "b5be34b68b235fd62797cb56a0946d3e"
server_key_version: "encryptionkey@0"
I20260811 03:04:40.731366 30999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "64e451830adc4cccabb39a37894d6ac0"
format_stamp: "Formatted at 2026-08-11 03:04:40 on dist-test-slave-8tdl"
server_key: "6859080740b25672ca3f245658f4074f"
server_key_iv: "b5be34b68b235fd62797cb56a0946d3e"
server_key_version: "encryptionkey@0"
I20260811 03:04:40.737283 30999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 03:04:40.742132 31015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:40.743791 30999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 03:04:40.744024 30999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "64e451830adc4cccabb39a37894d6ac0"
format_stamp: "Formatted at 2026-08-11 03:04:40 on dist-test-slave-8tdl"
server_key: "6859080740b25672ca3f245658f4074f"
server_key_iv: "b5be34b68b235fd62797cb56a0946d3e"
server_key_version: "encryptionkey@0"
I20260811 03:04:40.744280 30999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:40.764758 30999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:40.769832 30999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:40.770152 30999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:40.771392 30999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:40.773042 30999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:40.773129 30999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:40.773252 30999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:40.773303 30999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:40.813055 30999 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:38087
I20260811 03:04:40.813112 31142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:38087 every 8 connection(s)
I20260811 03:04:40.814581 30999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:40.818567 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 30999
I20260811 03:04:40.818877 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:40.819727 28084 external_mini_cluster.cc:1428] Setting key 4273222d6a987c58e0150e7c72de2d65
Aug 11 03:04:40 dist-test-slave-8tdl krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:40.838310 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:40.817481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44757 (local address 127.27.109.62:44945)
0811 03:04:40.818015 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:40.818020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:40.819484 (+ 1464us) server_negotiation.cc:407] Connection header received
0811 03:04:40.821772 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:40.821777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:40.821864 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:40.821995 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:40.823723 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:40.824510 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:40.826120 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:40.826531 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:40.826675 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:40.830585 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:40.830594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:40.830603 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:40.830652 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:40.834412 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:40.835406 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:40.835415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:40.835421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:40.835587 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:40.836056 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:40.836064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:40.836070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:40.836479 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:40.836648 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:40.837512 (+ 864us) server_negotiation.cc:299] Negotiation successful
0811 03:04:40.837799 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":239,"threads_started":1}
I20260811 03:04:40.839231 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:40.818189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44945 (local address 127.27.109.1:44757)
0811 03:04:40.819277 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:40.819318 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:40.821411 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:40.822266 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:40.822276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:40.822927 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:40.823538 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:40.823558 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:40.824693 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:40.824698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:40.825967 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:40.825980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:40.826809 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:40.826818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:40.827083 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:40.827820 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:40.827848 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:40.830327 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:40.834604 (+ 4277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:40.834612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:40.834628 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:40.835234 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:40.835711 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:40.835715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:40.835717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:40.835928 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:40.836628 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:40.836635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:40.837070 (+ 435us) client_negotiation.cc:769] Sending connection context
0811 03:04:40.837327 (+ 257us) client_negotiation.cc:240] Negotiation successful
0811 03:04:40.837787 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":481,"threads_started":1}
I20260811 03:04:40.841141 31143 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44945
I20260811 03:04:40.841583 31143 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:40.842451 31143 heartbeater.cc:511] Master 127.27.109.62:44945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260811 03:04:40.845039 30925 ts_manager.cc:194] Registered new tserver with Master: 64e451830adc4cccabb39a37894d6ac0 (127.27.109.1:38087)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
I20260811 03:04:40.846827 30925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:44757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:04:40.875571 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44945
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:41.244067 31151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:41.244411 31151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:41.244534 31151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:41.253633 31151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:41.253793 31151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:41.253937 31151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:04:41.265482 31151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:41.266930 31151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:41.268311 31151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:41.282539 31159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:41.282526 31160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:41.282557 31163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:41.283576 31151 server_base.cc:1034] running on GCE node
I20260811 03:04:41.284264 31151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:41.285342 31151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:41.286607 31151 hybrid_clock.cc:648] HybridClock initialized: now 1786417481286559 us; error 39 us; skew 500 ppm
I20260811 03:04:41.286988 31151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:41 dist-test-slave-8tdl krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:41.290580 31151 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:04:41.292263 31151 webserver.cc:460] Webserver started at http://127.27.109.2:34663/ using document root <none> and password file <none>
I20260811 03:04:41.292971 31151 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:41.293079 31151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:41.293424 31151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:41.296839 31151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "6a18bc421d0a454885494503b73b1f0e"
format_stamp: "Formatted at 2026-08-11 03:04:41 on dist-test-slave-8tdl"
server_key: "fa51129ab8c673eb2a82cbfbf6133659"
server_key_iv: "251b04fb1d8af81258884c76fa6ba413"
server_key_version: "encryptionkey@0"
I20260811 03:04:41.297780 31151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "6a18bc421d0a454885494503b73b1f0e"
format_stamp: "Formatted at 2026-08-11 03:04:41 on dist-test-slave-8tdl"
server_key: "fa51129ab8c673eb2a82cbfbf6133659"
server_key_iv: "251b04fb1d8af81258884c76fa6ba413"
server_key_version: "encryptionkey@0"
I20260811 03:04:41.304703 31151 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260811 03:04:41.309530 31171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:41.311414 31151 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 03:04:41.311636 31151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "6a18bc421d0a454885494503b73b1f0e"
format_stamp: "Formatted at 2026-08-11 03:04:41 on dist-test-slave-8tdl"
server_key: "fa51129ab8c673eb2a82cbfbf6133659"
server_key_iv: "251b04fb1d8af81258884c76fa6ba413"
server_key_version: "encryptionkey@0"
I20260811 03:04:41.311930 31151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:41.331741 31151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:41.337100 31151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:41.337431 31151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:41.338524 31151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:41.340305 31151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:41.340420 31151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:41.340572 31151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:41.340644 31151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:41.375982 31151 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:33113
I20260811 03:04:41.376055 31298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:33113 every 8 connection(s)
I20260811 03:04:41.377379 31151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:04:41.381906 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 31151
I20260811 03:04:41.382202 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:04:41.382654 28084 external_mini_cluster.cc:1428] Setting key d07b38b092ec59c100a8e1d1dc391c73
Aug 11 03:04:41 dist-test-slave-8tdl krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:41.402259 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:41.380236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:44449 (local address 127.27.109.62:44945)
0811 03:04:41.380882 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:41.380887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:41.381730 (+ 843us) server_negotiation.cc:407] Connection header received
0811 03:04:41.384056 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:41.384062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:41.384152 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:41.384287 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:41.386094 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:41.386987 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:41.388638 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:41.389148 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:41.389307 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:41.393829 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:41.393840 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:41.393848 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:41.393991 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:41.398191 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:41.399215 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:41.399224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:41.399230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:41.399418 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:41.399939 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:41.399947 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:41.399953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:41.400395 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:41.400566 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:41.401405 (+ 839us) server_negotiation.cc:299] Negotiation successful
0811 03:04:41.401713 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":356,"threads_started":1}
I20260811 03:04:41.403002 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:41.380698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44945 (local address 127.27.109.2:44449)
0811 03:04:41.381456 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:41.381503 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:41.383760 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:41.384625 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:41.384640 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:41.385286 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:41.385903 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:41.385923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:41.387166 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:41.387170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:41.388464 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:41.388477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:41.389415 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:41.389423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:41.389688 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:41.390462 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:41.390492 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:41.393503 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:41.398407 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:41.398416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:41.398432 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:41.399060 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:41.399569 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:41.399573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:41.399575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:41.399794 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:41.400531 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:41.400537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:41.400982 (+ 445us) client_negotiation.cc:769] Sending connection context
0811 03:04:41.401246 (+ 264us) client_negotiation.cc:240] Negotiation successful
0811 03:04:41.401698 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":307,"threads_started":1}
I20260811 03:04:41.404748 31299 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44945
I20260811 03:04:41.405092 31299 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:41.405866 31299 heartbeater.cc:511] Master 127.27.109.62:44945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260811 03:04:41.407634 30925 ts_manager.cc:194] Registered new tserver with Master: 6a18bc421d0a454885494503b73b1f0e (127.27.109.2:33113)
I20260811 03:04:41.408542 30925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:44449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:04:41.439715 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44945
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:41.788513 31308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:41.788856 31308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:41.789032 31308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:41.796931 31308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:41.797092 31308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:41.797325 31308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:04:41.808017 31308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:41.809508 31308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:41.810886 31308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:41.825558 31313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:41.825650 31315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:41.825625 31319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:41.826498 31308 server_base.cc:1034] running on GCE node
I20260811 03:04:41.827272 31308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:41.828318 31308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:41.829563 31308 hybrid_clock.cc:648] HybridClock initialized: now 1786417481829523 us; error 40 us; skew 500 ppm
I20260811 03:04:41.829918 31308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:41 dist-test-slave-8tdl krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:41.833427 31308 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:04:41.835160 31308 webserver.cc:460] Webserver started at http://127.27.109.3:42971/ using document root <none> and password file <none>
I20260811 03:04:41.835872 31308 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:41.835976 31308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:41.836252 31308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:41.839404 31308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "10140b4a79bb46deabedb78f6d11ce7c"
format_stamp: "Formatted at 2026-08-11 03:04:41 on dist-test-slave-8tdl"
server_key: "344f75eafdc0fe5fe3e9d9bde69a5004"
server_key_iv: "e31898f6e0a16f78e0f5b387aba8f82f"
server_key_version: "encryptionkey@0"
I20260811 03:04:41.840435 31308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "10140b4a79bb46deabedb78f6d11ce7c"
format_stamp: "Formatted at 2026-08-11 03:04:41 on dist-test-slave-8tdl"
server_key: "344f75eafdc0fe5fe3e9d9bde69a5004"
server_key_iv: "e31898f6e0a16f78e0f5b387aba8f82f"
server_key_version: "encryptionkey@0"
I20260811 03:04:41.846481 31308 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:04:41.850160 31143 heartbeater.cc:503] Master 127.27.109.62:44945 was elected leader, sending a full tablet report...
I20260811 03:04:41.850668 31328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:41.852519 31308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:04:41.852677 31308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "10140b4a79bb46deabedb78f6d11ce7c"
format_stamp: "Formatted at 2026-08-11 03:04:41 on dist-test-slave-8tdl"
server_key: "344f75eafdc0fe5fe3e9d9bde69a5004"
server_key_iv: "e31898f6e0a16f78e0f5b387aba8f82f"
server_key_version: "encryptionkey@0"
I20260811 03:04:41.852883 31308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:41.869630 31308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:41.874364 31308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:41.874642 31308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:41.875794 31308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:41.877241 31308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:41.877324 31308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:41.877508 31308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:41.877575 31308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:41.914417 31308 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:35047
I20260811 03:04:41.914459 31456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:35047 every 8 connection(s)
I20260811 03:04:41.916038 31308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:04:41.924202 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 31308
I20260811 03:04:41.924422 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:04:41.924851 28084 external_mini_cluster.cc:1428] Setting key 1e655fc0d7ead475c9c3f397ccb07a2e
Aug 11 03:04:41 dist-test-slave-8tdl krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:41.945782 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:41.919077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:48957 (local address 127.27.109.62:44945)
0811 03:04:41.919570 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:41.919575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:41.920863 (+ 1288us) server_negotiation.cc:407] Connection header received
0811 03:04:41.922892 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:41.922898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:41.923014 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:41.923155 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:41.925146 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:41.926210 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:41.928635 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:41.929090 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:41.929197 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:41.938275 (+ 9078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:41.938281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:41.938285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:41.938347 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:41.942250 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:41.943310 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:41.943315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:41.943318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:41.943461 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:41.943908 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:41.943933 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:41.943935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:41.944229 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:41.944350 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:41.945242 (+ 892us) server_negotiation.cc:299] Negotiation successful
0811 03:04:41.945427 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":263,"threads_started":1}
I20260811 03:04:41.946882 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:41.919647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44945 (local address 127.27.109.3:48957)
0811 03:04:41.920670 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:41.920710 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:41.922567 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:41.923414 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:41.923424 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:41.924131 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:41.924862 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:41.924884 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:41.926438 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:41.926446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:41.928463 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:41.928477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:41.929310 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:41.929316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:41.929542 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:41.934923 (+ 5381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:41.934956 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:41.937844 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:41.942441 (+ 4597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:41.942451 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:41.942469 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:41.943168 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:41.943585 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:41.943589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:41.943591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:41.943804 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:41.944352 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:41.944359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:41.944853 (+ 494us) client_negotiation.cc:769] Sending connection context
0811 03:04:41.945125 (+ 272us) client_negotiation.cc:240] Negotiation successful
0811 03:04:41.945573 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":567,"threads_started":1}
I20260811 03:04:41.948541 31458 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44945
I20260811 03:04:41.948858 31458 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:41.949663 31458 heartbeater.cc:511] Master 127.27.109.62:44945 requested a full tablet report, sending...
I20260811 03:04:41.951277 30925 ts_manager.cc:194] Registered new tserver with Master: 10140b4a79bb46deabedb78f6d11ce7c (127.27.109.3:35047)
I20260811 03:04:41.952066 30925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:48957
I20260811 03:04:41.954422 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 03:04:41.996675 31460 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:04:41.993722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.27.109.62:44945)
0811 03:04:41.994015 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:41.994023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:41.994386 (+ 363us) server_negotiation.cc:407] Connection header received
0811 03:04:41.994637 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:41.994641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:41.994770 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:41.994898 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:41.996355 (+ 1457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52084: BlockingRecv error: recv got EOF from 127.0.0.1:52084 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260811 03:04:42.000211 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30999
W20260811 03:04:42.003753 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30999: Not authorized: Client connection negotiation failed: client connection to 127.27.109.1:38087: server requires authentication, but client does not have Kerberos credentials available
I20260811 03:04:42.003960 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 30999
I20260811 03:04:42.026260 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31151
W20260811 03:04:42.029335 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31151: Not authorized: Client connection negotiation failed: client connection to 127.27.109.2:33113: server requires authentication, but client does not have Kerberos credentials available
I20260811 03:04:42.029515 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 31151
I20260811 03:04:42.051473 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31308
W20260811 03:04:42.054944 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31308: Not authorized: Client connection negotiation failed: client connection to 127.27.109.3:35047: server requires authentication, but client does not have Kerberos credentials available
I20260811 03:04:42.055174 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 31308
I20260811 03:04:42.076028 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 30891
I20260811 03:04:42.077576 30964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:42.207487 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 30891
2026-08-11T03:04:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4091 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:42 dist-test-slave-8tdl krb5kdc[31479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:42 dist-test-slave-8tdl krb5kdc[31479](info): set up 2 sockets
Aug 11 03:04:42 dist-test-slave-8tdl krb5kdc[31479](info): commencing operation
krb5kdc: starting...
W20260811 03:04:43.495220 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:43 dist-test-slave-8tdl krb5kdc[31479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:04:43.743209 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41451
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:44.097878 31494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:44.098192 31494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:44.098289 31494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:44.105772 31494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:44.105894 31494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:44.105954 31494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:44.106007 31494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:44.106050 31494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:44.116508 31494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:44.117828 31494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:44.119216 31494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:44.132458 31499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:44.132570 31500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:44.132603 31502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:44.133607 31494 server_base.cc:1034] running on GCE node
I20260811 03:04:44.134488 31494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:44.136148 31494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:44.137391 31494 hybrid_clock.cc:648] HybridClock initialized: now 1786417484137343 us; error 49 us; skew 500 ppm
I20260811 03:04:44.137789 31494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:44 dist-test-slave-8tdl krb5kdc[31479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417484, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:44.148013 31494 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:44.149885 31494 webserver.cc:460] Webserver started at http://127.27.109.62:44919/ using document root <none> and password file <none>
I20260811 03:04:44.150645 31494 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:44.150857 31494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:44.151199 31494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:44.154361 31494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "649faeaf572e466cafbe6c617c6ef0cc"
format_stamp: "Formatted at 2026-08-11 03:04:44 on dist-test-slave-8tdl"
server_key: "4ce189628e7176558347f288b0b35496"
server_key_iv: "6d4a01ac83c4aa851e4b7c2c77bf0d3b"
server_key_version: "encryptionkey@0"
I20260811 03:04:44.155323 31494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "649faeaf572e466cafbe6c617c6ef0cc"
format_stamp: "Formatted at 2026-08-11 03:04:44 on dist-test-slave-8tdl"
server_key: "4ce189628e7176558347f288b0b35496"
server_key_iv: "6d4a01ac83c4aa851e4b7c2c77bf0d3b"
server_key_version: "encryptionkey@0"
I20260811 03:04:44.160749 31494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:04:44.164721 31509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:44.166123 31494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:04:44.166326 31494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "649faeaf572e466cafbe6c617c6ef0cc"
format_stamp: "Formatted at 2026-08-11 03:04:44 on dist-test-slave-8tdl"
server_key: "4ce189628e7176558347f288b0b35496"
server_key_iv: "6d4a01ac83c4aa851e4b7c2c77bf0d3b"
server_key_version: "encryptionkey@0"
I20260811 03:04:44.166539 31494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:44.186120 31494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:44.190748 31494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:44.191073 31494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:44.211191 31494 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41451
I20260811 03:04:44.211249 31567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41451 every 8 connection(s)
I20260811 03:04:44.212661 31494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:44.216732 31568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:44.217911 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 31494
I20260811 03:04:44.218173 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:44.218947 28084 external_mini_cluster.cc:1428] Setting key 66cba348a45b5c7fa96dd8a29a997ebc
I20260811 03:04:44.232553 31568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc: Bootstrap starting.
Aug 11 03:04:44 dist-test-slave-8tdl krb5kdc[31479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:44.236858 31568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:44.238521 31568 log.cc:824] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:44.242177 31568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc: No bootstrap required, opened a new log
I20260811 03:04:44.245440 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:44.220787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.27.109.62:41451)
0811 03:04:44.221697 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:44.221709 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:44.221750 (+ 41us) server_negotiation.cc:407] Connection header received
0811 03:04:44.223497 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:44.223520 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:44.223912 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:44.224258 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:44.226323 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:44.227712 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:44.229804 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:44.230543 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:44.230767 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:44.235762 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:44.235775 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:44.235797 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:44.235858 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:44.240061 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:44.240931 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:44.240940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:44.240949 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:44.241184 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:44.241664 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:44.241670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:44.241673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:44.242496 (+ 823us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:44.242874 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:44.243423 (+ 549us) server_negotiation.cc:299] Negotiation successful
0811 03:04:44.243917 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":395,"threads_started":1}
I20260811 03:04:44.249655 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:04:44.252609 31568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649faeaf572e466cafbe6c617c6ef0cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41451 } }
I20260811 03:04:44.253063 31568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:44.253185 31568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649faeaf572e466cafbe6c617c6ef0cc, State: Initialized, Role: FOLLOWER
I20260811 03:04:44.253803 31568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649faeaf572e466cafbe6c617c6ef0cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41451 } }
I20260811 03:04:44.254148 31568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:44.254316 31568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:44.254578 31568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:44.257123 31568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649faeaf572e466cafbe6c617c6ef0cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41451 } }
I20260811 03:04:44.257805 31568 leader_election.cc:302] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 649faeaf572e466cafbe6c617c6ef0cc; no voters:
I20260811 03:04:44.258574 31568 leader_election.cc:288] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:44.258662 31574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:44.261255 31568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:44.263067 31574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [term 1 LEADER]: Becoming Leader. State: Replica: 649faeaf572e466cafbe6c617c6ef0cc, State: Running, Role: LEADER
I20260811 03:04:44.263641 31574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649faeaf572e466cafbe6c617c6ef0cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41451 } }
I20260811 03:04:44.266842 31575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "649faeaf572e466cafbe6c617c6ef0cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649faeaf572e466cafbe6c617c6ef0cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41451 } } }
I20260811 03:04:44.267194 31575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:44.266515 31576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 649faeaf572e466cafbe6c617c6ef0cc. Latest consensus state: current_term: 1 leader_uuid: "649faeaf572e466cafbe6c617c6ef0cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "649faeaf572e466cafbe6c617c6ef0cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41451 } } }
I20260811 03:04:44.269731 31576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:44.271678 31584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:44.277545 31584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:44.287201 31597 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 03:04:44.290773 31584 catalog_manager.cc:1347] Generated new cluster ID: 19e24ab94d414f00881c101ebe59ac1d
I20260811 03:04:44.290994 31584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:44.310624 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:44.301613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.27.109.62:41451)
0811 03:04:44.301890 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:44.301901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:44.302604 (+ 703us) server_negotiation.cc:407] Connection header received
0811 03:04:44.302892 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:44.302897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:44.302972 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:44.303104 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:44.304290 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:44.305158 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:44.306584 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:44.307228 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:44.307387 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:44.308694 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:44.308700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:44.308704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:04:44.309607 (+ 903us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:44.309708 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:04:44.309752 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:44.309895 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:44.310114 (+ 219us) server_negotiation.cc:299] Negotiation successful
0811 03:04:44.310226 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 03:04:44.311656 31584 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:44.313622 31584 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:44.323336 31584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 649faeaf572e466cafbe6c617c6ef0cc: Generated new TSK 0
I20260811 03:04:44.324371 31584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:04:44.342341 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31494
I20260811 03:04:44.343848 31563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:44.439426 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 31494
2026-08-11T03:04:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2225 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:44 dist-test-slave-8tdl krb5kdc[31609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:44 dist-test-slave-8tdl krb5kdc[31609](info): set up 2 sockets
Aug 11 03:04:44 dist-test-slave-8tdl krb5kdc[31609](info): commencing operation
krb5kdc: starting...
W20260811 03:04:45.597988 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:45 dist-test-slave-8tdl krb5kdc[31609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:04:45.847671 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37729
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:37729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:46.188546 31626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:46.188937 31626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:46.189110 31626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:46.196532 31626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:46.196705 31626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:46.196825 31626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:46.196931 31626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:46.197018 31626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:46.207509 31626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:37729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:46.209007 31626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:46.210413 31626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:46.223279 31631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:46.223227 31632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:46.223202 31634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:46.228219 31626 server_base.cc:1034] running on GCE node
I20260811 03:04:46.229143 31626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:46.230793 31626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:46.232036 31626 hybrid_clock.cc:648] HybridClock initialized: now 1786417486231962 us; error 59 us; skew 500 ppm
I20260811 03:04:46.232451 31626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:46 dist-test-slave-8tdl krb5kdc[31609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417486, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:46.236189 31626 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:46.237864 31626 webserver.cc:460] Webserver started at http://127.27.109.62:35801/ using document root <none> and password file <none>
I20260811 03:04:46.238659 31626 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:46.238881 31626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:46.239234 31626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:46.242589 31626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "7e41cf85bdb648b4b31c5919b185d951"
format_stamp: "Formatted at 2026-08-11 03:04:46 on dist-test-slave-8tdl"
server_key: "ba82bac9f8554b35c06c3fd7ae0aa33d"
server_key_iv: "57c92dc39dc9384692afc6a61a63c474"
server_key_version: "encryptionkey@0"
I20260811 03:04:46.243510 31626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "7e41cf85bdb648b4b31c5919b185d951"
format_stamp: "Formatted at 2026-08-11 03:04:46 on dist-test-slave-8tdl"
server_key: "ba82bac9f8554b35c06c3fd7ae0aa33d"
server_key_iv: "57c92dc39dc9384692afc6a61a63c474"
server_key_version: "encryptionkey@0"
I20260811 03:04:46.249122 31626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:04:46.252883 31641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:46.254227 31626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:04:46.254427 31626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "7e41cf85bdb648b4b31c5919b185d951"
format_stamp: "Formatted at 2026-08-11 03:04:46 on dist-test-slave-8tdl"
server_key: "ba82bac9f8554b35c06c3fd7ae0aa33d"
server_key_iv: "57c92dc39dc9384692afc6a61a63c474"
server_key_version: "encryptionkey@0"
I20260811 03:04:46.254643 31626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:46.271730 31626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:46.276185 31626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:46.276522 31626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:46.296710 31626 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:37729
I20260811 03:04:46.296749 31699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:37729 every 8 connection(s)
I20260811 03:04:46.298292 31626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:46.302217 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:46.302469 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 31626
I20260811 03:04:46.302744 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:46.303436 28084 external_mini_cluster.cc:1428] Setting key 90a890e3d27f611fea4615fd84208917
I20260811 03:04:46.316823 31700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951: Bootstrap starting.
Aug 11 03:04:46 dist-test-slave-8tdl krb5kdc[31609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:46.320865 31700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:46.322463 31700 log.cc:824] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:46.326046 31700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951: No bootstrap required, opened a new log
I20260811 03:04:46.329227 31703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:46.305502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.27.109.62:37729)
0811 03:04:46.306326 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:46.306338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:46.306382 (+ 44us) server_negotiation.cc:407] Connection header received
0811 03:04:46.308459 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:46.308483 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:46.308967 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:46.309337 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:46.311229 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:46.312552 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:46.314461 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:46.315338 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:46.315500 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:46.320350 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:46.320359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:46.320375 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:46.320418 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:46.324725 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:46.325508 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:46.325517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:46.325527 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:46.325721 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:46.326157 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:46.326163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:46.326166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:46.326821 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:46.327157 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:46.327639 (+ 482us) server_negotiation.cc:299] Negotiation successful
0811 03:04:46.328122 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":364,"threads_started":1}
I20260811 03:04:46.334470 31700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e41cf85bdb648b4b31c5919b185d951" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37729 } }
I20260811 03:04:46.334980 31700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:46.335099 31700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e41cf85bdb648b4b31c5919b185d951, State: Initialized, Role: FOLLOWER
I20260811 03:04:46.335752 31700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e41cf85bdb648b4b31c5919b185d951" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37729 } }
I20260811 03:04:46.336057 31700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:46.336277 31700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:46.336477 31700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:46.338966 31700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e41cf85bdb648b4b31c5919b185d951" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37729 } }
I20260811 03:04:46.339596 31700 leader_election.cc:302] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e41cf85bdb648b4b31c5919b185d951; no voters:
I20260811 03:04:46.340327 31700 leader_election.cc:288] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:46.340416 31705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:46.341216 31705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [term 1 LEADER]: Becoming Leader. State: Replica: 7e41cf85bdb648b4b31c5919b185d951, State: Running, Role: LEADER
I20260811 03:04:46.341790 31705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e41cf85bdb648b4b31c5919b185d951" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37729 } }
I20260811 03:04:46.342569 31700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:46.344908 31707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e41cf85bdb648b4b31c5919b185d951. Latest consensus state: current_term: 1 leader_uuid: "7e41cf85bdb648b4b31c5919b185d951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e41cf85bdb648b4b31c5919b185d951" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37729 } } }
I20260811 03:04:46.345289 31707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:46.346257 31706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e41cf85bdb648b4b31c5919b185d951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e41cf85bdb648b4b31c5919b185d951" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37729 } } }
I20260811 03:04:46.346743 31706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951 [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:46.348176 31711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:46.354144 31711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:46.363562 31711 catalog_manager.cc:1347] Generated new cluster ID: ad6cc5aa94524dd298b6bb94ee227644
I20260811 03:04:46.363699 31711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:46.395625 31711 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:46.397070 31711 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:46.410121 31711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e41cf85bdb648b4b31c5919b185d951: Generated new TSK 0
I20260811 03:04:46.411175 31711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:04:46.468573 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37729
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:46.824944 31727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:46.825275 31727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:46.825376 31727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:46.832886 31727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:46.833019 31727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:46.833139 31727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:46.843951 31727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:46.845386 31727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:46.846741 31727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:46.861212 31733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:46.861313 31736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:46.861258 31732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:46.862066 31727 server_base.cc:1034] running on GCE node
I20260811 03:04:46.862819 31727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:46.863916 31727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:46.865168 31727 hybrid_clock.cc:648] HybridClock initialized: now 1786417486865110 us; error 51 us; skew 500 ppm
I20260811 03:04:46.865525 31727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:46 dist-test-slave-8tdl krb5kdc[31609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417486, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:46.869002 31727 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:04:46.870649 31727 webserver.cc:460] Webserver started at http://127.27.109.1:36051/ using document root <none> and password file <none>
I20260811 03:04:46.871522 31727 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:46.871675 31727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:46.872000 31727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:46.875166 31727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "dacea71481324d21a8326a786325dfc5"
format_stamp: "Formatted at 2026-08-11 03:04:46 on dist-test-slave-8tdl"
server_key: "80bb3ff70b6f7b24f9a0a3b12848f9a3"
server_key_iv: "96df9146c5ec1181f5e8bf2d6e770400"
server_key_version: "encryptionkey@0"
I20260811 03:04:46.875996 31727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "dacea71481324d21a8326a786325dfc5"
format_stamp: "Formatted at 2026-08-11 03:04:46 on dist-test-slave-8tdl"
server_key: "80bb3ff70b6f7b24f9a0a3b12848f9a3"
server_key_iv: "96df9146c5ec1181f5e8bf2d6e770400"
server_key_version: "encryptionkey@0"
I20260811 03:04:46.881460 31727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:04:46.885113 31746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:46.886493 31727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 03:04:46.886674 31727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "dacea71481324d21a8326a786325dfc5"
format_stamp: "Formatted at 2026-08-11 03:04:46 on dist-test-slave-8tdl"
server_key: "80bb3ff70b6f7b24f9a0a3b12848f9a3"
server_key_iv: "96df9146c5ec1181f5e8bf2d6e770400"
server_key_version: "encryptionkey@0"
I20260811 03:04:46.886929 31727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:46.898833 31727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:46.903194 31727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:46.903524 31727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:46.904544 31727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:46.905946 31727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:46.906023 31727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:46.906193 31727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:46.906261 31727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:46.942328 31727 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:46695
I20260811 03:04:46.942427 31876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:46695 every 8 connection(s)
I20260811 03:04:46.943861 31727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:46.953501 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 31727
I20260811 03:04:46.953775 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:46.954567 28084 external_mini_cluster.cc:1428] Setting key aa9115dd2145510ed38a899b0262d389
Aug 11 03:04:46 dist-test-slave-8tdl krb5kdc[31609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417486, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:46.966410 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:46.947328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:43767 (local address 127.27.109.62:37729)
0811 03:04:46.947870 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:46.947875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:46.948904 (+ 1029us) server_negotiation.cc:407] Connection header received
0811 03:04:46.950889 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:46.950894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:46.950983 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:46.951112 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:46.952708 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:46.953464 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:46.955339 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:46.955830 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:46.955951 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:46.960243 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:46.960252 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:46.960257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:46.960325 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:46.963180 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:46.963965 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:46.963970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:46.963974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:46.964106 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:46.964566 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:46.964571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:46.964580 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:46.964846 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:46.965008 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:46.965768 (+ 760us) server_negotiation.cc:299] Negotiation successful
0811 03:04:46.966019 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":261,"threads_started":1}
I20260811 03:04:46.967594 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:46.947958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37729 (local address 127.27.109.1:43767)
0811 03:04:46.948717 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:46.948757 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:46.950541 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:46.951378 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:46.951386 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:46.951909 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:46.952533 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:46.952551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:46.953646 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:46.953651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:46.955142 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:46.955155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:46.956082 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:46.956091 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:46.956383 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:46.957238 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:46.957261 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:46.959907 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:46.963351 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:46.963361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:46.963381 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:46.963825 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:46.964222 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:46.964227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:46.964231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:46.964437 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:46.964989 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:46.964998 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:46.965371 (+ 373us) client_negotiation.cc:769] Sending connection context
0811 03:04:46.965649 (+ 278us) client_negotiation.cc:240] Negotiation successful
0811 03:04:46.966270 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":311,"threads_started":1}
I20260811 03:04:46.969385 31877 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37729
I20260811 03:04:46.969736 31877 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:46.970538 31877 heartbeater.cc:511] Master 127.27.109.62:37729 requested a full tablet report, sending...
I20260811 03:04:46.972749 31662 ts_manager.cc:194] Registered new tserver with Master: dacea71481324d21a8326a786325dfc5 (127.27.109.1:46695)
I20260811 03:04:46.974262 31662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:43767
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:04:47.014788 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37729
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:47.366521 31885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:47.366961 31885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:47.367066 31885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:47.374454 31885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:47.374594 31885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:47.374794 31885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:04:47.385269 31885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:47.386659 31885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:47.388243 31885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:47.402782 31894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:47.402824 31892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:47.402737 31891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:47.404098 31885 server_base.cc:1034] running on GCE node
I20260811 03:04:47.404811 31885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:47.405875 31885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:47.407083 31885 hybrid_clock.cc:648] HybridClock initialized: now 1786417487407025 us; error 42 us; skew 500 ppm
I20260811 03:04:47.407436 31885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:47 dist-test-slave-8tdl krb5kdc[31609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417487, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:47.410990 31885 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:04:47.412710 31885 webserver.cc:460] Webserver started at http://127.27.109.2:43061/ using document root <none> and password file <none>
I20260811 03:04:47.413528 31885 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:47.413697 31885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:47.414057 31885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:47.417570 31885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "731043a7c38c4b60a5c72ba20c6d7853"
format_stamp: "Formatted at 2026-08-11 03:04:47 on dist-test-slave-8tdl"
server_key: "1ba3189c833ff4d3435e9b48982a739b"
server_key_iv: "22df240a289fda9b17d4c933412c68a2"
server_key_version: "encryptionkey@0"
I20260811 03:04:47.418597 31885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "731043a7c38c4b60a5c72ba20c6d7853"
format_stamp: "Formatted at 2026-08-11 03:04:47 on dist-test-slave-8tdl"
server_key: "1ba3189c833ff4d3435e9b48982a739b"
server_key_iv: "22df240a289fda9b17d4c933412c68a2"
server_key_version: "encryptionkey@0"
I20260811 03:04:47.424623 31885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260811 03:04:47.428754 31902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:47.430334 31885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:04:47.430572 31885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "731043a7c38c4b60a5c72ba20c6d7853"
format_stamp: "Formatted at 2026-08-11 03:04:47 on dist-test-slave-8tdl"
server_key: "1ba3189c833ff4d3435e9b48982a739b"
server_key_iv: "22df240a289fda9b17d4c933412c68a2"
server_key_version: "encryptionkey@0"
I20260811 03:04:47.430886 31885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:47.459851 31885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:47.464811 31885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:47.465189 31885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:47.466326 31885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:47.467895 31885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:47.468024 31885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:47.468190 31885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:47.468271 31885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:47.506079 31885 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45577
I20260811 03:04:47.506124 32029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45577 every 8 connection(s)
I20260811 03:04:47.507639 31885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:04:47.510080 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 31885
I20260811 03:04:47.510305 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:04:47.510987 28084 external_mini_cluster.cc:1428] Setting key 318932b6a915def96974b162b20059b1
Aug 11 03:04:47 dist-test-slave-8tdl krb5kdc[31609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417487, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:47.536201 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:47.510612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39633 (local address 127.27.109.62:37729)
0811 03:04:47.511154 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:47.511161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:47.512500 (+ 1339us) server_negotiation.cc:407] Connection header received
0811 03:04:47.515569 (+ 3069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:47.515574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:47.515650 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:47.515736 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:47.518250 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:47.519113 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:47.522033 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:47.522850 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:47.522999 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:47.527575 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:47.527586 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:47.527595 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:47.527652 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:47.532062 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:47.532965 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:47.532975 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:47.532982 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:47.533182 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:47.533726 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:47.533732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:47.533736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:47.534207 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:47.534436 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:47.535440 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0811 03:04:47.535725 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":286,"threads_started":1}
I20260811 03:04:47.537371 32035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:47.511346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37729 (local address 127.27.109.2:39633)
0811 03:04:47.512317 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:47.512369 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:47.515185 (+ 2816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:47.516075 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:47.516092 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:47.517016 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:47.517992 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:47.518026 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:47.519297 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:47.519306 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:47.521824 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:47.521850 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:47.523131 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:47.523137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:47.523376 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:47.524220 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:47.524247 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:47.527264 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:47.532267 (+ 5003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:47.532275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:47.532291 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:47.532811 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:47.533360 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:47.533364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:47.533367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:47.533591 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:47.534427 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:47.534434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:47.534957 (+ 523us) client_negotiation.cc:769] Sending connection context
0811 03:04:47.535331 (+ 374us) client_negotiation.cc:240] Negotiation successful
0811 03:04:47.535974 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":425,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260811 03:04:47.539179 32030 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37729
I20260811 03:04:47.539561 32030 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:47.540449 32030 heartbeater.cc:511] Master 127.27.109.62:37729 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260811 03:04:47.542351 31662 ts_manager.cc:194] Registered new tserver with Master: 731043a7c38c4b60a5c72ba20c6d7853 (127.27.109.2:45577)
I20260811 03:04:47.543512 31662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:39633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:04:47.571355 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37729
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:47.937848 32038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:47.938166 32038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:47.938277 32038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:47.950004 32038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:47.950142 32038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:47.950274 32038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:04:47.963323 32038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:47.964808 32038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:47.966188 32038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 03:04:47.977342 31877 heartbeater.cc:503] Master 127.27.109.62:37729 was elected leader, sending a full tablet report...
W20260811 03:04:47.980702 32044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:47.980581 32045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:47.981364 32047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:47.982360 32038 server_base.cc:1034] running on GCE node
I20260811 03:04:47.983172 32038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:47.984313 32038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:47.985586 32038 hybrid_clock.cc:648] HybridClock initialized: now 1786417487985531 us; error 45 us; skew 500 ppm
I20260811 03:04:47.986048 32038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:47 dist-test-slave-8tdl krb5kdc[31609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417487, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:47.989740 32038 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:04:47.992025 32038 webserver.cc:460] Webserver started at http://127.27.109.3:36621/ using document root <none> and password file <none>
I20260811 03:04:47.992830 32038 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:47.993059 32038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:47.993429 32038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:47.996833 32038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "236ef01d04784d3f8c35d14a97d95f36"
format_stamp: "Formatted at 2026-08-11 03:04:47 on dist-test-slave-8tdl"
server_key: "d603daaa69076f8c406f385d3467ba7c"
server_key_iv: "03e82f53b75405d4cb2f3bb312d1bf37"
server_key_version: "encryptionkey@0"
I20260811 03:04:47.997732 32038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "236ef01d04784d3f8c35d14a97d95f36"
format_stamp: "Formatted at 2026-08-11 03:04:47 on dist-test-slave-8tdl"
server_key: "d603daaa69076f8c406f385d3467ba7c"
server_key_iv: "03e82f53b75405d4cb2f3bb312d1bf37"
server_key_version: "encryptionkey@0"
I20260811 03:04:48.004078 32038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260811 03:04:48.008807 32055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:48.011075 32038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 03:04:48.011317 32038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "236ef01d04784d3f8c35d14a97d95f36"
format_stamp: "Formatted at 2026-08-11 03:04:47 on dist-test-slave-8tdl"
server_key: "d603daaa69076f8c406f385d3467ba7c"
server_key_iv: "03e82f53b75405d4cb2f3bb312d1bf37"
server_key_version: "encryptionkey@0"
I20260811 03:04:48.011585 32038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:48.033938 32038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:48.039448 32038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:48.039798 32038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:48.040973 32038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:48.042536 32038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:48.042645 32038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:48.042810 32038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:48.042891 32038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:48.080559 32038 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:33789
I20260811 03:04:48.080608 32186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:33789 every 8 connection(s)
I20260811 03:04:48.082007 32038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:04:48.088204 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 32038
I20260811 03:04:48.088457 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:04:48.089058 28084 external_mini_cluster.cc:1428] Setting key fc29f080432d45a66a4512771e4d9056
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[31609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417487, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:48.107321 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:48.084997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:39581 (local address 127.27.109.62:37729)
0811 03:04:48.085504 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:48.085509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:48.086760 (+ 1251us) server_negotiation.cc:407] Connection header received
0811 03:04:48.089644 (+ 2884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:48.089649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:48.089779 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:48.089912 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:48.092467 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.093409 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.095353 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.095830 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.095962 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:48.100169 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:48.100175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:48.100179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:48.100220 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:48.103207 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.104351 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.104358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.104362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.104547 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.105143 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.105147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.105149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.105599 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:48.105804 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:48.106780 (+ 976us) server_negotiation.cc:299] Negotiation successful
0811 03:04:48.107034 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":253,"threads_started":1}
I20260811 03:04:48.108232 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:48.085665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37729 (local address 127.27.109.3:39581)
0811 03:04:48.086538 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:48.086591 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:48.089231 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:48.090313 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:48.090330 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:48.091379 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:48.092275 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:48.092297 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.093587 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.093591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:48.095177 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:48.095192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.096149 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.096154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:48.096374 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:48.097188 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:48.097219 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:48.099873 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:48.103397 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:48.103410 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:48.103434 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:48.104142 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:48.104686 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:48.104691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:48.104694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:48.105001 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:48.105799 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:48.105806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:48.106267 (+ 461us) client_negotiation.cc:769] Sending connection context
0811 03:04:48.106555 (+ 288us) client_negotiation.cc:240] Negotiation successful
0811 03:04:48.107015 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":390,"threads_started":1}
I20260811 03:04:48.109851 32187 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37729
I20260811 03:04:48.110184 32187 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:48.111086 32187 heartbeater.cc:511] Master 127.27.109.62:37729 requested a full tablet report, sending...
I20260811 03:04:48.112703 31662 ts_manager.cc:194] Registered new tserver with Master: 236ef01d04784d3f8c35d14a97d95f36 (127.27.109.3:33789)
I20260811 03:04:48.113533 31662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:39581
I20260811 03:04:48.118927 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[31609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417488, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[31609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417488, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:48.181802 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:48.166102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.27.109.62:37729)
0811 03:04:48.166287 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:48.166292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:48.166902 (+ 610us) server_negotiation.cc:407] Connection header received
0811 03:04:48.167025 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:48.167031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:48.167117 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:48.167226 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:48.168961 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.169736 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.171297 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.171726 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.171854 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:48.176067 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:48.176073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:48.176077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:48.176118 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:48.179053 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.179778 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.179782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.179784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.179919 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.180440 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.180444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.180446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.180812 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:48.181068 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:48.181376 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 03:04:48.181571 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260811 03:04:48.182533 31662 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54232
I20260811 03:04:48.184839 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31727
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[31609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417488, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:48.204490 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:48.186217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.27.109.1:46695)
0811 03:04:48.187014 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:48.187023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:48.187049 (+ 26us) server_negotiation.cc:407] Connection header received
0811 03:04:48.187112 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:48.187119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:48.187347 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:48.187548 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:48.189082 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.190318 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.191982 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.192656 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.192825 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:48.197641 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:48.197658 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:48.197676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:48.197724 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:48.201177 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.202072 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.202080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.202082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.202233 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.202718 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.202729 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.202733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.203136 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:48.203491 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:48.203801 (+ 310us) server_negotiation.cc:299] Negotiation successful
0811 03:04:48.204114 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":364,"threads_started":1}
W20260811 03:04:48.205979 31870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41594
W20260811 03:04:48.207348 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31727: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:48.207633 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 31727
I20260811 03:04:48.230778 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31885
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[31609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417488, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:04:48.251407 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:48.231770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.27.109.2:45577)
0811 03:04:48.232434 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:48.232442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:48.232476 (+ 34us) server_negotiation.cc:407] Connection header received
0811 03:04:48.232541 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:48.232547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:48.232772 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:48.232901 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:48.234415 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.235750 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.237389 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.238075 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.238221 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:48.243823 (+ 5602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:48.243840 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:48.243856 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:48.243901 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:48.247575 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.248712 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.248721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.248724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.248897 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.249495 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.249499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.249507 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.249972 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:48.250258 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:48.250743 (+ 485us) server_negotiation.cc:299] Negotiation successful
0811 03:04:48.251077 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":360,"threads_started":1}
W20260811 03:04:48.252790 32024 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52116
W20260811 03:04:48.254045 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31885: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:48.254241 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 31885
I20260811 03:04:48.275609 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 32038
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[31609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417488, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:04:48.295327 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:48.276824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.27.109.3:33789)
0811 03:04:48.277796 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:48.277804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:48.277838 (+ 34us) server_negotiation.cc:407] Connection header received
0811 03:04:48.277912 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:48.277919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:48.278150 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:48.278304 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:48.279850 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.280949 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.282510 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:48.283362 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:48.283478 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:48.288639 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:48.288656 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:48.288672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:48.288715 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:48.292268 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.293084 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.293092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.293095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.293250 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:48.293664 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:48.293668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:48.293670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:48.294119 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:48.294384 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:48.294664 (+ 280us) server_negotiation.cc:299] Negotiation successful
0811 03:04:48.295026 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":634,"threads_started":1}
W20260811 03:04:48.296970 32181 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45632
W20260811 03:04:48.298295 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 32038: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:04:48.298482 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 32038
I20260811 03:04:48.325757 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 31626
I20260811 03:04:48.327203 31694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:48.439424 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 31626
2026-08-11T03:04:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4006 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[32206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[32206](info): set up 2 sockets
Aug 11 03:04:48 dist-test-slave-8tdl krb5kdc[32206](info): commencing operation
krb5kdc: starting...
W20260811 03:04:49.631166 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:49 dist-test-slave-8tdl krb5kdc[32206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
W20260811 03:04:50.172127 28084 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.109.62: real 0.115s user 0.002s sys 0.036s
I20260811 03:04:50.175971 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37049
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:37049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:51.092496 32220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:51.093107 32220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:51.093310 32220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:51.101750 32220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:51.101938 32220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:51.102058 32220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:51.102176 32220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:51.102258 32220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:51.113642 32220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:37049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:51.115300 32220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:51.116886 32220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:51.130800 32227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:51.130749 32226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:51.131102 32229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:51.131851 32220 server_base.cc:1034] running on GCE node
I20260811 03:04:51.132769 32220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:51.134518 32220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:51.135774 32220 hybrid_clock.cc:648] HybridClock initialized: now 1786417491135715 us; error 43 us; skew 500 ppm
I20260811 03:04:51.136291 32220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:51 dist-test-slave-8tdl krb5kdc[32206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:51.145246 32220 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:51.147487 32220 webserver.cc:460] Webserver started at http://127.27.109.62:39947/ using document root <none> and password file <none>
I20260811 03:04:51.148413 32220 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:51.148619 32220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:51.149029 32220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:51.153133 32220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "b5586e7846ab4884a62680dba89f27a3"
format_stamp: "Formatted at 2026-08-11 03:04:51 on dist-test-slave-8tdl"
server_key: "7a78ec834e1871d9ebd4ec3a9aaf0458"
server_key_iv: "2b8516e058557db47163f0dd0b7f1fa5"
server_key_version: "encryptionkey@0"
I20260811 03:04:51.154604 32220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "b5586e7846ab4884a62680dba89f27a3"
format_stamp: "Formatted at 2026-08-11 03:04:51 on dist-test-slave-8tdl"
server_key: "7a78ec834e1871d9ebd4ec3a9aaf0458"
server_key_iv: "2b8516e058557db47163f0dd0b7f1fa5"
server_key_version: "encryptionkey@0"
I20260811 03:04:51.161492 32220 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:04:51.165928 32237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:51.167770 32220 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:04:51.168032 32220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "b5586e7846ab4884a62680dba89f27a3"
format_stamp: "Formatted at 2026-08-11 03:04:51 on dist-test-slave-8tdl"
server_key: "7a78ec834e1871d9ebd4ec3a9aaf0458"
server_key_iv: "2b8516e058557db47163f0dd0b7f1fa5"
server_key_version: "encryptionkey@0"
I20260811 03:04:51.168301 32220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:51.186395 32220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:51.191916 32220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:51.192256 32220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:51.213672 32220 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:37049
I20260811 03:04:51.213708 32300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:37049 every 8 connection(s)
I20260811 03:04:51.215394 32220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:51.220484 32301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:51.222144 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 32220
I20260811 03:04:51.222396 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:51.223361 28084 external_mini_cluster.cc:1428] Setting key 5052c6a964325bf3c1fec610b0852e72
I20260811 03:04:51.237462 32301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3: Bootstrap starting.
Aug 11 03:04:51 dist-test-slave-8tdl krb5kdc[32206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:51.241897 32301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:51.243633 32301 log.cc:824] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:51.248076 32301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3: No bootstrap required, opened a new log
I20260811 03:04:51.251117 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:51.225849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.27.109.62:37049)
0811 03:04:51.226847 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:51.226865 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:51.226927 (+ 62us) server_negotiation.cc:407] Connection header received
0811 03:04:51.228682 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:51.228711 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:51.229214 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:51.229599 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:51.231603 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:51.233066 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:51.235037 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:51.235805 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:51.235953 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:51.241251 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:51.241260 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:51.241278 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:51.241331 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:51.245464 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:51.246384 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:51.246394 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:51.246403 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:51.246754 (+ 351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:51.247260 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:51.247267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:51.247270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:51.248075 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:51.248380 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:51.248918 (+ 538us) server_negotiation.cc:299] Negotiation successful
0811 03:04:51.249466 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":464,"threads_started":1}
I20260811 03:04:51.257560 32301 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5586e7846ab4884a62680dba89f27a3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37049 } }
I20260811 03:04:51.258059 32301 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:51.258201 32301 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5586e7846ab4884a62680dba89f27a3, State: Initialized, Role: FOLLOWER
I20260811 03:04:51.258965 32301 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5586e7846ab4884a62680dba89f27a3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37049 } }
I20260811 03:04:51.259248 32301 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:51.259389 32301 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:51.259572 32301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:51.261900 32301 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5586e7846ab4884a62680dba89f27a3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37049 } }
I20260811 03:04:51.262533 32301 leader_election.cc:302] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5586e7846ab4884a62680dba89f27a3; no voters:
I20260811 03:04:51.263218 32301 leader_election.cc:288] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:51.263391 32306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:51.264149 32306 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [term 1 LEADER]: Becoming Leader. State: Replica: b5586e7846ab4884a62680dba89f27a3, State: Running, Role: LEADER
I20260811 03:04:51.264816 32306 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5586e7846ab4884a62680dba89f27a3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37049 } }
I20260811 03:04:51.265616 32301 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:51.267993 32308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5586e7846ab4884a62680dba89f27a3. Latest consensus state: current_term: 1 leader_uuid: "b5586e7846ab4884a62680dba89f27a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5586e7846ab4884a62680dba89f27a3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37049 } } }
I20260811 03:04:51.268160 32307 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5586e7846ab4884a62680dba89f27a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5586e7846ab4884a62680dba89f27a3" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37049 } } }
I20260811 03:04:51.268520 32308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:51.268577 32307 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3 [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:51.271027 32314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:51.276458 32314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:51.286830 32314 catalog_manager.cc:1347] Generated new cluster ID: 2a10511bde5b459fbb5ef409ebe3881a
I20260811 03:04:51.286989 32314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:51.317606 32314 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:51.319876 32314 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:51.335181 32314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5586e7846ab4884a62680dba89f27a3: Generated new TSK 0
I20260811 03:04:51.336432 32314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:04:51.411579 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37049
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:51.800602 32330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:51.800946 32330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:51.801131 32330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:51.809227 32330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:51.809479 32330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:51.809686 32330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:51.820843 32330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:51.822443 32330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:51.824150 32330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:51.840430 32338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:51.840541 32336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:51.840477 32335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:51.845456 32330 server_base.cc:1034] running on GCE node
I20260811 03:04:51.846482 32330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:51.847879 32330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:51.849175 32330 hybrid_clock.cc:648] HybridClock initialized: now 1786417491849118 us; error 60 us; skew 500 ppm
I20260811 03:04:51.849603 32330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:51 dist-test-slave-8tdl krb5kdc[32206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:51.853742 32330 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:04:51.855933 32330 webserver.cc:460] Webserver started at http://127.27.109.1:44353/ using document root <none> and password file <none>
I20260811 03:04:51.857211 32330 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:51.857438 32330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:51.857828 32330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:51.861438 32330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "e81aeea133c5452e935de0d6eee3f171"
format_stamp: "Formatted at 2026-08-11 03:04:51 on dist-test-slave-8tdl"
server_key: "677dd432e63fce59fdd7f5d02510cf29"
server_key_iv: "460242a1f766f1aaf24205273740c6a4"
server_key_version: "encryptionkey@0"
I20260811 03:04:51.862470 32330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "e81aeea133c5452e935de0d6eee3f171"
format_stamp: "Formatted at 2026-08-11 03:04:51 on dist-test-slave-8tdl"
server_key: "677dd432e63fce59fdd7f5d02510cf29"
server_key_iv: "460242a1f766f1aaf24205273740c6a4"
server_key_version: "encryptionkey@0"
I20260811 03:04:51.869239 32330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:04:51.874073 32346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:51.875967 32330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:04:51.876251 32330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "e81aeea133c5452e935de0d6eee3f171"
format_stamp: "Formatted at 2026-08-11 03:04:51 on dist-test-slave-8tdl"
server_key: "677dd432e63fce59fdd7f5d02510cf29"
server_key_iv: "460242a1f766f1aaf24205273740c6a4"
server_key_version: "encryptionkey@0"
I20260811 03:04:51.876574 32330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:51.899417 32330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:51.905316 32330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:51.905707 32330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:51.906941 32330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:51.908610 32330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:51.908732 32330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:51.908931 32330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:51.909009 32330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:51.949740 32330 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:43865
I20260811 03:04:51.949785 32472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:43865 every 8 connection(s)
I20260811 03:04:51.951406 32330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:51.961350 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 32330
I20260811 03:04:51.961596 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:51.962661 28084 external_mini_cluster.cc:1428] Setting key 4d57fe18cc15e473d7fddffa0f3ae503
Aug 11 03:04:51 dist-test-slave-8tdl krb5kdc[32206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417491, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:51.978235 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:51.954630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:48067 (local address 127.27.109.62:37049)
0811 03:04:51.955244 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:51.955249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:51.956409 (+ 1160us) server_negotiation.cc:407] Connection header received
0811 03:04:51.958580 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:51.958586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:51.958749 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:51.958878 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:51.960735 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:51.961563 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:51.963869 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:51.964338 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:51.964502 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:51.969388 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:51.969399 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:51.969407 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:51.969469 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:51.973961 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:51.975262 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:51.975267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:51.975269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:51.975434 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:51.976104 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:51.976108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:51.976110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:51.976474 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:51.976613 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:51.977731 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0811 03:04:51.977959 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":319,"threads_started":1}
I20260811 03:04:51.979980 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:51.955407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37049 (local address 127.27.109.1:48067)
0811 03:04:51.956213 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:51.956261 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:51.958215 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:51.959208 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:51.959218 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:51.959841 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:51.960510 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:51.960532 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:51.961765 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:51.961770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:51.963677 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:51.963699 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:51.964639 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:51.964648 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:51.964991 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:51.965947 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:51.965982 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:51.968982 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:51.974264 (+ 5282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:51.974274 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:51.974291 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:51.975054 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:51.975569 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:51.975576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:51.975581 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:51.975906 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:51.976639 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:51.976651 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:51.977191 (+ 540us) client_negotiation.cc:769] Sending connection context
0811 03:04:51.977586 (+ 395us) client_negotiation.cc:240] Negotiation successful
0811 03:04:51.978255 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":316,"threads_started":1}
I20260811 03:04:51.982040 32473 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37049
I20260811 03:04:51.982543 32473 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:51.983469 32473 heartbeater.cc:511] Master 127.27.109.62:37049 requested a full tablet report, sending...
I20260811 03:04:51.985970 32258 ts_manager.cc:194] Registered new tserver with Master: e81aeea133c5452e935de0d6eee3f171 (127.27.109.1:43865)
I20260811 03:04:51.987697 32258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:48067
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:04:52.028141 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37049
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:52.427752 32479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:52.428115 32479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:52.428256 32479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:52.436594 32479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:52.436741 32479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:52.436873 32479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:04:52.447875 32479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:52.449352 32479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:52.450838 32479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:52.466125 32486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:52.466187 32488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:52.466457 32485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:52.467814 32479 server_base.cc:1034] running on GCE node
I20260811 03:04:52.468538 32479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:52.469743 32479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:52.471027 32479 hybrid_clock.cc:648] HybridClock initialized: now 1786417492470936 us; error 78 us; skew 500 ppm
I20260811 03:04:52.471379 32479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:52 dist-test-slave-8tdl krb5kdc[32206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:52.475440 32479 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:04:52.477293 32479 webserver.cc:460] Webserver started at http://127.27.109.2:36867/ using document root <none> and password file <none>
I20260811 03:04:52.478260 32479 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:52.478459 32479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:52.479001 32479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:52.482990 32479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "6892ae0cd1df4de08aa2ffb824147018"
format_stamp: "Formatted at 2026-08-11 03:04:52 on dist-test-slave-8tdl"
server_key: "b4c7f52c24597ceb2b9f12ea6c1fd3ad"
server_key_iv: "133a0d92322a406d900711ff97c6f26a"
server_key_version: "encryptionkey@0"
I20260811 03:04:52.484256 32479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "6892ae0cd1df4de08aa2ffb824147018"
format_stamp: "Formatted at 2026-08-11 03:04:52 on dist-test-slave-8tdl"
server_key: "b4c7f52c24597ceb2b9f12ea6c1fd3ad"
server_key_iv: "133a0d92322a406d900711ff97c6f26a"
server_key_version: "encryptionkey@0"
I20260811 03:04:52.491351 32479 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260811 03:04:52.496053 32495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:52.498201 32479 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:04:52.498447 32479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "6892ae0cd1df4de08aa2ffb824147018"
format_stamp: "Formatted at 2026-08-11 03:04:52 on dist-test-slave-8tdl"
server_key: "b4c7f52c24597ceb2b9f12ea6c1fd3ad"
server_key_iv: "133a0d92322a406d900711ff97c6f26a"
server_key_version: "encryptionkey@0"
I20260811 03:04:52.498751 32479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:52.515887 32479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:52.521355 32479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:52.521737 32479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:52.523190 32479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:52.524813 32479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:52.524932 32479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:52.525112 32479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:52.525207 32479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:52.564280 32479 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:36097
I20260811 03:04:52.564386 32616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:36097 every 8 connection(s)
I20260811 03:04:52.565832 32479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:04:52.567031 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 32479
I20260811 03:04:52.567253 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:04:52.567858 28084 external_mini_cluster.cc:1428] Setting key 9eeddf060e7356c101b538c04635f987
Aug 11 03:04:52 dist-test-slave-8tdl krb5kdc[32206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417492, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:52.592141 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:52.569309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:33723 (local address 127.27.109.62:37049)
0811 03:04:52.569920 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:52.569927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:52.571318 (+ 1391us) server_negotiation.cc:407] Connection header received
0811 03:04:52.573609 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:52.573619 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:52.573734 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:52.573859 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:52.575898 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:52.577281 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:52.579200 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:52.580084 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:52.580210 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:52.585331 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:52.585338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:52.585342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:52.585387 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:52.588423 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:52.589363 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:52.589369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:52.589371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:52.589515 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:52.590040 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:52.590044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:52.590046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:52.590373 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:52.590544 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:52.591446 (+ 902us) server_negotiation.cc:299] Negotiation successful
0811 03:04:52.591736 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":312,"threads_started":1}
I20260811 03:04:52.593264 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:52.570093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37049 (local address 127.27.109.2:33723)
0811 03:04:52.571092 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:52.571166 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:52.573218 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:52.574200 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:52.574211 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:52.574955 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:52.575655 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:52.575678 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:52.577516 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:52.577521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:52.579011 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:52.579028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:52.580394 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:52.580404 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:52.580880 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:52.581794 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:52.581833 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:52.585004 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:52.588642 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:52.588653 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:52.588678 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:52.589191 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:52.589646 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:52.589651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:52.589655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:52.589895 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:52.590519 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:52.590528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:52.591015 (+ 487us) client_negotiation.cc:769] Sending connection context
0811 03:04:52.591324 (+ 309us) client_negotiation.cc:240] Negotiation successful
0811 03:04:52.591971 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":378,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260811 03:04:52.595054 32617 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37049
I20260811 03:04:52.595403 32617 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:52.596477 32617 heartbeater.cc:511] Master 127.27.109.62:37049 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260811 03:04:52.598475 32258 ts_manager.cc:194] Registered new tserver with Master: 6892ae0cd1df4de08aa2ffb824147018 (127.27.109.2:36097)
I20260811 03:04:52.599418 32258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:33723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:04:52.630539 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37049
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:04:52.991425 32473 heartbeater.cc:503] Master 127.27.109.62:37049 was elected leader, sending a full tablet report...
W20260811 03:04:53.016444 32623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:53.016927 32623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:53.017117 32623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:53.025136 32623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:53.025280 32623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:53.025429 32623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:04:53.036473 32623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:53.037984 32623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:53.039642 32623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:53.054998 32628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:53.054917 32630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:53.054973 32632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:53.055963 32623 server_base.cc:1034] running on GCE node
I20260811 03:04:53.056690 32623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:53.057775 32623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:53.059032 32623 hybrid_clock.cc:648] HybridClock initialized: now 1786417493058963 us; error 40 us; skew 500 ppm
I20260811 03:04:53.059448 32623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[32206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:53.063453 32623 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:04:53.065205 32623 webserver.cc:460] Webserver started at http://127.27.109.3:44795/ using document root <none> and password file <none>
I20260811 03:04:53.066082 32623 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:53.066255 32623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:53.066615 32623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:53.070161 32623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "33113acd364b49f18ff1e593f924e98a"
format_stamp: "Formatted at 2026-08-11 03:04:53 on dist-test-slave-8tdl"
server_key: "3927a76482023e862d91fdfb726f1e78"
server_key_iv: "ed8ce320297c16c0105c94f00ca9548b"
server_key_version: "encryptionkey@0"
I20260811 03:04:53.071205 32623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "33113acd364b49f18ff1e593f924e98a"
format_stamp: "Formatted at 2026-08-11 03:04:53 on dist-test-slave-8tdl"
server_key: "3927a76482023e862d91fdfb726f1e78"
server_key_iv: "ed8ce320297c16c0105c94f00ca9548b"
server_key_version: "encryptionkey@0"
I20260811 03:04:53.078055 32623 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260811 03:04:53.082401 32640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:53.084334 32623 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:04:53.084584 32623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "33113acd364b49f18ff1e593f924e98a"
format_stamp: "Formatted at 2026-08-11 03:04:53 on dist-test-slave-8tdl"
server_key: "3927a76482023e862d91fdfb726f1e78"
server_key_iv: "ed8ce320297c16c0105c94f00ca9548b"
server_key_version: "encryptionkey@0"
I20260811 03:04:53.084826 32623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:53.102154 32623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:53.108047 32623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:53.108445 32623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:53.109687 32623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:53.111478 32623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:53.111581 32623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:53.111769 32623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:53.111846 32623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:53.152558 32623 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:36135
I20260811 03:04:53.152613 300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:36135 every 8 connection(s)
I20260811 03:04:53.154249 32623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:04:53.158627 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 32623
I20260811 03:04:53.158905 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:04:53.159632 28084 external_mini_cluster.cc:1428] Setting key 130d8d4ea82814ac07bbd7d158453452
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[32206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417493, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:53.181234 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:53.157497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:34035 (local address 127.27.109.62:37049)
0811 03:04:53.158260 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:53.158266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:53.159682 (+ 1416us) server_negotiation.cc:407] Connection header received
0811 03:04:53.162188 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:53.162193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:53.162293 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:53.162416 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:53.164728 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.165780 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.167880 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.168354 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.168517 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:53.173150 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:53.173156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:53.173159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:53.173204 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:53.177096 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.178154 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.178159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.178162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.178315 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.178990 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.178995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.178997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.179341 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:53.179537 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:53.180450 (+ 913us) server_negotiation.cc:299] Negotiation successful
0811 03:04:53.180755 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":281,"threads_started":1}
I20260811 03:04:53.182507 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:53.158370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37049 (local address 127.27.109.3:34035)
0811 03:04:53.159487 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:53.159539 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:53.161870 (+ 2331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:53.162777 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:53.162787 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:53.163776 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:53.164502 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:53.164526 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.165995 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.166007 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:53.167666 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:53.167683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.168649 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.168659 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:53.168962 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:53.169899 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:53.169929 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:53.172834 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:53.177307 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:53.177325 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:53.177346 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:53.177978 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:53.178460 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:53.178468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:53.178474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:53.178839 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:53.179497 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:53.179507 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:53.180004 (+ 497us) client_negotiation.cc:769] Sending connection context
0811 03:04:53.180315 (+ 311us) client_negotiation.cc:240] Negotiation successful
0811 03:04:53.181052 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":558,"threads_started":1}
I20260811 03:04:53.184985 302 heartbeater.cc:347] Connected to a master server at 127.27.109.62:37049
I20260811 03:04:53.185395 302 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:53.186303 302 heartbeater.cc:511] Master 127.27.109.62:37049 requested a full tablet report, sending...
I20260811 03:04:53.188390 32256 ts_manager.cc:194] Registered new tserver with Master: 33113acd364b49f18ff1e593f924e98a (127.27.109.3:36135)
I20260811 03:04:53.190104 32256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:34035
I20260811 03:04:53.191208 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[32206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[32206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:53.254439 310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:53.237261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.27.109.1:43865)
0811 03:04:53.237969 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:53.237976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:53.238013 (+ 37us) server_negotiation.cc:407] Connection header received
0811 03:04:53.238177 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:53.238188 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:53.238446 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:53.238601 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:53.240346 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.241545 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.243046 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.243661 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.243784 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:53.248255 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:53.248281 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:53.248298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:53.248347 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:53.251567 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.252429 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.252436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.252439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.252619 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.252995 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.252998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.253000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.253375 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:53.253642 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:53.253905 (+ 263us) server_negotiation.cc:299] Negotiation successful
0811 03:04:53.254137 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":321,"threads_started":1}
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[32206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:53.276400 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:53.259936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.27.109.62:37049)
0811 03:04:53.260182 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:53.260189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:53.260311 (+ 122us) server_negotiation.cc:407] Connection header received
0811 03:04:53.260493 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:53.260496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:53.260587 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:53.260676 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:53.262321 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.263306 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.264595 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.265100 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.265209 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:53.270017 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:53.270028 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:53.270035 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:53.270092 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:53.273244 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.274209 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.274214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.274216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.274364 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.274816 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.274820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.274822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.275226 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:53.275474 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:53.275823 (+ 349us) server_negotiation.cc:299] Negotiation successful
0811 03:04:53.276036 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260811 03:04:53.277124 32258 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44984
I20260811 03:04:53.278375 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 32330
I20260811 03:04:53.293797 310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:53.279364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.27.109.1:43865)
0811 03:04:53.279548 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:53.279553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:53.279913 (+ 360us) server_negotiation.cc:407] Connection header received
0811 03:04:53.280077 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:53.280081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:53.280181 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:53.280267 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:53.281697 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.282511 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.284160 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.284877 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.284991 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:53.287970 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:53.287977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:53.287980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:53.288023 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:53.291056 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.291822 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.291827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.291829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.291983 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.292338 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.292341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.292344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.292603 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:53.292796 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:53.293283 (+ 487us) server_negotiation.cc:299] Negotiation successful
0811 03:04:53.293512 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 03:04:53.294664 32468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:53.385444 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 32330
I20260811 03:04:53.410372 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 32479
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[32206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:04:53.429281 317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:53.411620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.27.109.2:36097)
0811 03:04:53.412219 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:53.412227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:53.412256 (+ 29us) server_negotiation.cc:407] Connection header received
0811 03:04:53.412324 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:53.412331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:53.412581 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:53.412771 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:53.414186 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.415525 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.416925 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.417556 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.417714 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:53.422424 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:53.422442 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:53.422459 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:53.422514 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:53.425834 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.426731 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.426744 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.426751 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.426937 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.427381 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.427387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.427391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.427801 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:53.428164 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:53.428557 (+ 393us) server_negotiation.cc:299] Negotiation successful
0811 03:04:53.428870 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":300,"threads_started":1}
I20260811 03:04:53.430809 32612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:53.518826 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 32479
I20260811 03:04:53.541659 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 32623
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[32206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:04:53.561621 319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:53.543083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.27.109.3:36135)
0811 03:04:53.543752 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:53.543759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:53.543792 (+ 33us) server_negotiation.cc:407] Connection header received
0811 03:04:53.543858 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:53.543866 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:53.544122 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:53.544316 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:53.545755 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.546921 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.548308 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:53.548919 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:53.549339 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:53.554604 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:53.554620 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:53.554638 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:53.554984 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:53.558336 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.559313 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.559321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.559324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.559482 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:53.559920 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:53.559924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:53.559927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:53.560401 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:53.560682 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:53.560989 (+ 307us) server_negotiation.cc:299] Negotiation successful
0811 03:04:53.561299 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":298,"threads_started":1}
I20260811 03:04:53.563315 32763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:53.657183 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 32623
I20260811 03:04:53.679157 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 32220
I20260811 03:04:53.680523 32296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:53.795594 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 32220
2026-08-11T03:04:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5360 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[324](info): set up 2 sockets
Aug 11 03:04:53 dist-test-slave-8tdl krb5kdc[324](info): commencing operation
krb5kdc: starting...
W20260811 03:04:54.992475 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:04:55 dist-test-slave-8tdl krb5kdc[324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:04:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:04:55.267611 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38595
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:55.643407 338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:55.643748 338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:55.643847 338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:55.651430 338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:55.651566 338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:55.651613 338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:55.651674 338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:55.651726 338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:55.662199 338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:55.663744 338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:55.665269 338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:55.678382 344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:55.678555 348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:55.678450 345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:55.680055 338 server_base.cc:1034] running on GCE node
I20260811 03:04:55.681046 338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:55.682835 338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:55.684125 338 hybrid_clock.cc:648] HybridClock initialized: now 1786417495684061 us; error 58 us; skew 500 ppm
I20260811 03:04:55.684576 338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:55 dist-test-slave-8tdl krb5kdc[324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417495, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:55.688398 338 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:04:55.689908 338 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:55.690063 338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:55.690428 338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:55.693827 338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "9574efc4787e494587d60e77d55dc8ba"
format_stamp: "Formatted at 2026-08-11 03:04:55 on dist-test-slave-8tdl"
server_key: "48d5198562ba088cd9aa065fffdf4a43"
server_key_iv: "28b7b2c5824ca22141b76c65971615b4"
server_key_version: "encryptionkey@0"
I20260811 03:04:55.694885 338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "9574efc4787e494587d60e77d55dc8ba"
format_stamp: "Formatted at 2026-08-11 03:04:55 on dist-test-slave-8tdl"
server_key: "48d5198562ba088cd9aa065fffdf4a43"
server_key_iv: "28b7b2c5824ca22141b76c65971615b4"
server_key_version: "encryptionkey@0"
I20260811 03:04:55.700939 338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:04:55.705150 355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:55.706805 338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:04:55.707096 338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "9574efc4787e494587d60e77d55dc8ba"
format_stamp: "Formatted at 2026-08-11 03:04:55 on dist-test-slave-8tdl"
server_key: "48d5198562ba088cd9aa065fffdf4a43"
server_key_iv: "28b7b2c5824ca22141b76c65971615b4"
server_key_version: "encryptionkey@0"
I20260811 03:04:55.707365 338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:55.731812 338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:55.737591 338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:55.737926 338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:55.759079 338 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:38595
I20260811 03:04:55.759115 409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:38595 every 8 connection(s)
I20260811 03:04:55.760172 338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:55.764674 411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:55.765219 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 338
I20260811 03:04:55.765443 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:55.766176 28084 external_mini_cluster.cc:1428] Setting key 62ff33af489022a6f3802c75d5f56069
I20260811 03:04:55.781934 411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba: Bootstrap starting.
Aug 11 03:04:55 dist-test-slave-8tdl krb5kdc[324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:55.786996 411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:55.788834 411 log.cc:824] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:55.793254 411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba: No bootstrap required, opened a new log
I20260811 03:04:55.796732 416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:55.768629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.27.109.62:38595)
0811 03:04:55.769606 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:55.769619 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:55.769665 (+ 46us) server_negotiation.cc:407] Connection header received
0811 03:04:55.771891 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:55.771921 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:55.772413 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:55.772819 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:55.775301 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:55.776746 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:55.779032 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:55.779924 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:55.780117 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:55.785872 (+ 5755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:55.785887 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:55.785911 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:55.785980 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:55.790723 (+ 4743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:55.791741 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:55.791751 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:55.791761 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:55.792034 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:55.792523 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:55.792529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:55.792532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:55.793450 (+ 918us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:55.793846 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:55.794328 (+ 482us) server_negotiation.cc:299] Negotiation successful
0811 03:04:55.794950 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":439,"threads_started":1}
I20260811 03:04:55.804474 411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574efc4787e494587d60e77d55dc8ba" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38595 } }
I20260811 03:04:55.804968 411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:55.805109 411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9574efc4787e494587d60e77d55dc8ba, State: Initialized, Role: FOLLOWER
I20260811 03:04:55.805859 411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574efc4787e494587d60e77d55dc8ba" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38595 } }
I20260811 03:04:55.806134 411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:55.806299 411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:55.806488 411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:55.809180 411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574efc4787e494587d60e77d55dc8ba" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38595 } }
I20260811 03:04:55.809810 411 leader_election.cc:302] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9574efc4787e494587d60e77d55dc8ba; no voters:
I20260811 03:04:55.810475 411 leader_election.cc:288] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:55.810720 419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:55.811614 419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [term 1 LEADER]: Becoming Leader. State: Replica: 9574efc4787e494587d60e77d55dc8ba, State: Running, Role: LEADER
I20260811 03:04:55.812265 419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574efc4787e494587d60e77d55dc8ba" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38595 } }
I20260811 03:04:55.812992 411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:55.815734 421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9574efc4787e494587d60e77d55dc8ba. Latest consensus state: current_term: 1 leader_uuid: "9574efc4787e494587d60e77d55dc8ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574efc4787e494587d60e77d55dc8ba" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38595 } } }
I20260811 03:04:55.815680 420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9574efc4787e494587d60e77d55dc8ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9574efc4787e494587d60e77d55dc8ba" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38595 } } }
I20260811 03:04:55.816124 421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:55.816224 420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:55.817159 425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:55.825007 425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:55.839634 425 catalog_manager.cc:1347] Generated new cluster ID: f1e03a739d8a47f989c24475a209df2f
I20260811 03:04:55.839784 425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:55.873346 425 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:55.875375 425 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:55.883299 425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba: Generated new TSK 0
I20260811 03:04:55.884457 425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:04:55.963892 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38595
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:56.333647 445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:56.334028 445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:56.334201 445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:56.342162 445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:56.342365 445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:56.342555 445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:56.353440 445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:56.355042 445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:56.356499 445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:56.372277 450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:56.372252 454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:56.372203 451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:56.373140 445 server_base.cc:1034] running on GCE node
I20260811 03:04:56.373946 445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:56.375363 445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:56.376619 445 hybrid_clock.cc:648] HybridClock initialized: now 1786417496376552 us; error 61 us; skew 500 ppm
I20260811 03:04:56.377035 445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:56 dist-test-slave-8tdl krb5kdc[324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417496, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:56.385360 445 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:04:56.386747 445 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:56.386862 445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:56.387187 445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:56.390554 445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "a46371fb0e0e494eba42fc42c25f5930"
format_stamp: "Formatted at 2026-08-11 03:04:56 on dist-test-slave-8tdl"
server_key: "ac45e11151ae464d8ee5d20cee4295ea"
server_key_iv: "9bc912f02d6171c0fba62d9c5c02a076"
server_key_version: "encryptionkey@0"
I20260811 03:04:56.391532 445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "a46371fb0e0e494eba42fc42c25f5930"
format_stamp: "Formatted at 2026-08-11 03:04:56 on dist-test-slave-8tdl"
server_key: "ac45e11151ae464d8ee5d20cee4295ea"
server_key_iv: "9bc912f02d6171c0fba62d9c5c02a076"
server_key_version: "encryptionkey@0"
I20260811 03:04:56.397986 445 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260811 03:04:56.402252 460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:56.404083 445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260811 03:04:56.404302 445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "a46371fb0e0e494eba42fc42c25f5930"
format_stamp: "Formatted at 2026-08-11 03:04:56 on dist-test-slave-8tdl"
server_key: "ac45e11151ae464d8ee5d20cee4295ea"
server_key_iv: "9bc912f02d6171c0fba62d9c5c02a076"
server_key_version: "encryptionkey@0"
I20260811 03:04:56.404578 445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:56.441174 445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:56.446151 445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:56.446463 445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:56.447685 445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:56.449187 445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:56.449293 445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:56.449375 445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:56.449432 445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:56.485960 445 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:35375
I20260811 03:04:56.486013 591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:35375 every 8 connection(s)
I20260811 03:04:56.487018 445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:56.493173 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 445
I20260811 03:04:56.493480 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:56.494217 28084 external_mini_cluster.cc:1428] Setting key 866fcb3b7b846c67a4cff826c468bfc0
Aug 11 03:04:56 dist-test-slave-8tdl krb5kdc[324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417496, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:56.511416 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:56.490215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:59807 (local address 127.27.109.62:38595)
0811 03:04:56.490741 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:56.490748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:56.491925 (+ 1177us) server_negotiation.cc:407] Connection header received
0811 03:04:56.494212 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:56.494218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:56.494336 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:56.494470 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:56.496445 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:56.497191 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:56.498904 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:56.499326 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:56.499468 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:56.503614 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:56.503624 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:56.503631 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:56.503685 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:56.507027 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:56.508267 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:56.508275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:56.508279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:56.508514 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:56.509149 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:56.509155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:56.509158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:56.509563 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:56.509689 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:56.510812 (+ 1123us) server_negotiation.cc:299] Negotiation successful
0811 03:04:56.511099 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260811 03:04:56.513016 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:56.490959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38595 (local address 127.27.109.1:59807)
0811 03:04:56.491747 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:56.491790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:56.493904 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:56.494788 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:56.494798 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:56.495387 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:56.496258 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:56.496294 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:56.497350 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:56.497361 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:56.498751 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:56.498765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:56.499601 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:56.499615 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:56.499848 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:56.500637 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:56.500663 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:56.503308 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:56.507222 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:56.507237 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:56.507260 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:56.508013 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:56.508654 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:56.508662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:56.508668 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:56.508964 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:56.509709 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:56.509721 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:56.510250 (+ 529us) client_negotiation.cc:769] Sending connection context
0811 03:04:56.510601 (+ 351us) client_negotiation.cc:240] Negotiation successful
0811 03:04:56.511330 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":328,"threads_started":1}
I20260811 03:04:56.515231 592 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38595
I20260811 03:04:56.515612 592 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:56.516384 592 heartbeater.cc:511] Master 127.27.109.62:38595 requested a full tablet report, sending...
I20260811 03:04:56.518826 373 ts_manager.cc:194] Registered new tserver with Master: a46371fb0e0e494eba42fc42c25f5930 (127.27.109.1:35375)
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260811 03:04:56.520488 373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:59807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:04:56.555454 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38595
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:04:56.922104 600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:56.922473 600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:56.922652 600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:56.930470 600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:56.930644 600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:56.930840 600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:04:56.941612 600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:56.943181 600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:56.944643 600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:56.958732 606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:56.959144 610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:56.959182 607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:56.960579 600 server_base.cc:1034] running on GCE node
I20260811 03:04:56.961393 600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:56.962661 600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:56.963974 600 hybrid_clock.cc:648] HybridClock initialized: now 1786417496963896 us; error 76 us; skew 500 ppm
I20260811 03:04:56.964313 600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:56 dist-test-slave-8tdl krb5kdc[324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417496, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:56.968135 600 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:04:56.969616 600 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:56.969735 600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:56.970068 600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:56.973673 600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "d258cdc26fec4dee863a1bde73222fae"
format_stamp: "Formatted at 2026-08-11 03:04:56 on dist-test-slave-8tdl"
server_key: "4b65014d3bc38a2f49f346c3ffb25cc8"
server_key_iv: "6be164ef1289d376d9e82bc3d985a396"
server_key_version: "encryptionkey@0"
I20260811 03:04:56.974654 600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "d258cdc26fec4dee863a1bde73222fae"
format_stamp: "Formatted at 2026-08-11 03:04:56 on dist-test-slave-8tdl"
server_key: "4b65014d3bc38a2f49f346c3ffb25cc8"
server_key_iv: "6be164ef1289d376d9e82bc3d985a396"
server_key_version: "encryptionkey@0"
I20260811 03:04:56.980886 600 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260811 03:04:56.985148 616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:56.986987 600 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:04:56.987187 600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "d258cdc26fec4dee863a1bde73222fae"
format_stamp: "Formatted at 2026-08-11 03:04:56 on dist-test-slave-8tdl"
server_key: "4b65014d3bc38a2f49f346c3ffb25cc8"
server_key_iv: "6be164ef1289d376d9e82bc3d985a396"
server_key_version: "encryptionkey@0"
I20260811 03:04:56.987440 600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:57.024344 600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:57.030560 600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:57.031037 600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:57.032400 600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:57.034310 600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:57.034427 600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:57.034592 600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:57.034749 600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260811 03:04:57.076041 600 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:40367
I20260811 03:04:57.076067 747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:40367 every 8 connection(s)
I20260811 03:04:57.077049 600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:04:57.083611 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 600
I20260811 03:04:57.083813 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:04:57.084421 28084 external_mini_cluster.cc:1428] Setting key 614f2b6711e9a00563d96ce9d59876e2
Aug 11 03:04:57 dist-test-slave-8tdl krb5kdc[324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417496, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:57.106316 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.080368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:51609 (local address 127.27.109.62:38595)
0811 03:04:57.081039 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:57.081045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:57.082275 (+ 1230us) server_negotiation.cc:407] Connection header received
0811 03:04:57.085154 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:57.085162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:57.085266 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:57.085383 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:57.088085 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.088853 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.092273 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.092883 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.093017 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.099088 (+ 6071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:57.099095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:57.099099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:57.099151 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:57.102358 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:57.103485 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:57.103491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:57.103493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:57.103645 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:57.104192 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:57.104197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:57.104199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:57.104558 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:57.104744 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:57.105614 (+ 870us) server_negotiation.cc:299] Negotiation successful
0811 03:04:57.105905 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":301,"threads_started":1}
I20260811 03:04:57.107699 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.081154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38595 (local address 127.27.109.2:51609)
0811 03:04:57.082069 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:57.082132 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:57.084707 (+ 2575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:57.085770 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:57.085781 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:57.086582 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:57.087769 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.087821 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.089058 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.089068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.091936 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.091965 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.093201 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.093215 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.093648 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.094712 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:57.094764 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:57.098627 (+ 3863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:57.102589 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:57.102601 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:57.102625 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:57.103303 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:57.103787 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:57.103793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:57.103797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:57.104041 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:57.104714 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:57.104724 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:57.105170 (+ 446us) client_negotiation.cc:769] Sending connection context
0811 03:04:57.105495 (+ 325us) client_negotiation.cc:240] Negotiation successful
0811 03:04:57.106173 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":331,"threads_started":1}
I20260811 03:04:57.109606 748 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38595
I20260811 03:04:57.109982 748 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:57.110831 748 heartbeater.cc:511] Master 127.27.109.62:38595 requested a full tablet report, sending...
I20260811 03:04:57.112803 373 ts_manager.cc:194] Registered new tserver with Master: d258cdc26fec4dee863a1bde73222fae (127.27.109.2:40367)
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260811 03:04:57.113933 373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:51609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:04:57.151010 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38595
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:04:57.523890 592 heartbeater.cc:503] Master 127.27.109.62:38595 was elected leader, sending a full tablet report...
W20260811 03:04:57.528369 755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:57.528731 755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:57.528939 755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:57.536790 755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:57.536931 755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:57.537101 755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:04:57.547433 755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:57.548933 755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:57.550413 755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:57.564471 761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:57.564536 764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:57.564553 766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:57.565678 755 server_base.cc:1034] running on GCE node
I20260811 03:04:57.566476 755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:57.567824 755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:57.569098 755 hybrid_clock.cc:648] HybridClock initialized: now 1786417497569031 us; error 64 us; skew 500 ppm
I20260811 03:04:57.569517 755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:04:57 dist-test-slave-8tdl krb5kdc[324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:04:57.573357 755 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:04:57.574787 755 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:57.574942 755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:57.575307 755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:57.578650 755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "312aca42d7be43619da0f6d9a3beec6c"
format_stamp: "Formatted at 2026-08-11 03:04:57 on dist-test-slave-8tdl"
server_key: "4d9f59b74f841c046efee212e790ab06"
server_key_iv: "8ff62a49d44cdbd906ea5a550c0afcbe"
server_key_version: "encryptionkey@0"
I20260811 03:04:57.579682 755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "312aca42d7be43619da0f6d9a3beec6c"
format_stamp: "Formatted at 2026-08-11 03:04:57 on dist-test-slave-8tdl"
server_key: "4d9f59b74f841c046efee212e790ab06"
server_key_iv: "8ff62a49d44cdbd906ea5a550c0afcbe"
server_key_version: "encryptionkey@0"
I20260811 03:04:57.585525 755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 03:04:57.589663 772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:57.591408 755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:04:57.591614 755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "312aca42d7be43619da0f6d9a3beec6c"
format_stamp: "Formatted at 2026-08-11 03:04:57 on dist-test-slave-8tdl"
server_key: "4d9f59b74f841c046efee212e790ab06"
server_key_iv: "8ff62a49d44cdbd906ea5a550c0afcbe"
server_key_version: "encryptionkey@0"
I20260811 03:04:57.591879 755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:57.604247 755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:57.609113 755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:57.609488 755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:57.610502 755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:57.612093 755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:57.612218 755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:57.612358 755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:57.612447 755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:57.648460 755 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:35727
I20260811 03:04:57.648562 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:35727 every 8 connection(s)
I20260811 03:04:57.649441 755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:04:57.658955 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 755
I20260811 03:04:57.659160 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:04:57.659648 28084 external_mini_cluster.cc:1428] Setting key 67b5739d65ae362e44d4c838cdba812c
Aug 11 03:04:57 dist-test-slave-8tdl krb5kdc[324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417497, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:04:57.672264 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.652355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:54175 (local address 127.27.109.62:38595)
0811 03:04:57.652848 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:57.652853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:57.654002 (+ 1149us) server_negotiation.cc:407] Connection header received
0811 03:04:57.656042 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:57.656047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:57.656127 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:57.656243 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:57.657862 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.658957 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.661530 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.662139 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.662269 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.666392 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:57.666397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:57.666401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:57.666437 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:57.669205 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:57.670020 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:57.670023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:57.670026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:57.670170 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:57.670594 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:57.670597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:57.670599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:57.670930 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:57.671054 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:57.671755 (+ 701us) server_negotiation.cc:299] Negotiation successful
0811 03:04:57.671950 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
I20260811 03:04:57.673357 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.653005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38595 (local address 127.27.109.3:54175)
0811 03:04:57.653781 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:57.653821 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:57.655668 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:57.656467 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:57.656476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:57.657056 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:57.657672 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.657691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.659214 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.659219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.661215 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.661241 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.662437 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.662449 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.662791 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.663754 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:04:57.663780 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:57.666159 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:57.669371 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:57.669378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:57.669393 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:57.669886 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:57.670277 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:04:57.670281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:04:57.670286 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:04:57.670495 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:04:57.671054 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:57.671060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:57.671420 (+ 360us) client_negotiation.cc:769] Sending connection context
0811 03:04:57.671672 (+ 252us) client_negotiation.cc:240] Negotiation successful
0811 03:04:57.672144 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":307,"threads_started":1}
I20260811 03:04:57.675252 897 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38595
I20260811 03:04:57.675604 897 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:57.676509 897 heartbeater.cc:511] Master 127.27.109.62:38595 requested a full tablet report, sending...
I20260811 03:04:57.678047 373 ts_manager.cc:194] Registered new tserver with Master: 312aca42d7be43619da0f6d9a3beec6c (127.27.109.3:35727)
I20260811 03:04:57.678918 373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:54175
I20260811 03:04:57.689463 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:04:57.709327 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.694499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.27.109.62:38595)
0811 03:04:57.694743 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:57.694748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:57.695333 (+ 585us) server_negotiation.cc:407] Connection header received
0811 03:04:57.695513 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:57.695519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:57.695609 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:57.695704 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:57.697475 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.698403 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.699847 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.700309 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.700424 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.703269 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:57.703276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:57.703279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:57.703320 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:57.706110 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:57.706928 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:57.706933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:57.706936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:57.707091 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:57.707500 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:57.707503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:57.707506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:57.707967 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:57.708265 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:57.708692 (+ 427us) server_negotiation.cc:299] Negotiation successful
0811 03:04:57.708891 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 03:04:57.714252 373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:04:57.722561 373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:04:57.763289 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.737730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35375 (local address 127.0.0.1:49658)
0811 03:04:57.745178 (+ 7448us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:57.745267 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:57.745567 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:57.754192 (+ 8625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:57.754197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:57.754233 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:57.754810 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.754827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.757164 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.757170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.759945 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.759967 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.762041 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.762048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.762285 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.762368 (+ 83us) client_negotiation.cc:769] Sending connection context
0811 03:04:57.762506 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 03:04:57.762735 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7022,"mutex_wait_us":19,"thread_start_us":317,"threads_started":1}
I20260811 03:04:57.763353 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.738802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:35727 (local address 127.0.0.1:58412)
0811 03:04:57.752614 (+ 13812us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:57.752644 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:57.752862 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:57.753715 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:57.753726 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:57.753775 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:57.754379 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.754400 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.757151 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.757156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.760033 (+ 2877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.760049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.762353 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.762360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.762561 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.762594 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 03:04:57.762726 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 03:04:57.762869 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13614,"mutex_wait_us":70,"thread_start_us":13403,"threads_started":1}
I20260811 03:04:57.764429 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.738982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.27.109.3:35727)
0811 03:04:57.751945 (+ 12963us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:57.751958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:57.752829 (+ 871us) server_negotiation.cc:407] Connection header received
0811 03:04:57.753070 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:57.753089 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:57.753410 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:57.753646 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:57.754629 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.756982 (+ 2353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.760239 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.761823 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.761957 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.762201 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:57.763887 (+ 1686us) server_negotiation.cc:299] Negotiation successful
0811 03:04:57.764096 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12706,"thread_start_us":14020,"threads_started":1}
I20260811 03:04:57.764652 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.738044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.27.109.1:35375)
0811 03:04:57.753322 (+ 15278us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:57.753330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:57.753359 (+ 29us) server_negotiation.cc:407] Connection header received
0811 03:04:57.753469 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:57.753477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:57.753734 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:57.755062 (+ 1328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:57.755321 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.757001 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.760194 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.761792 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.763802 (+ 2010us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.764066 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:57.764151 (+ 85us) server_negotiation.cc:299] Negotiation successful
0811 03:04:57.764336 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15099,"thread_start_us":15848,"threads_started":1}
I20260811 03:04:57.766307 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.738950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:40367 (local address 127.0.0.1:38724)
0811 03:04:57.754408 (+ 15458us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:57.754434 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:57.754571 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:57.755212 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:57.755217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:57.755245 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:57.755718 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.755731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.757200 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.757207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.761570 (+ 4363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.761597 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.765425 (+ 3828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.765434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.765632 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.765671 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 03:04:57.765775 (+ 104us) client_negotiation.cc:240] Negotiation successful
0811 03:04:57.765923 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15310,"mutex_wait_us":56,"thread_start_us":15738,"threads_started":1}
I20260811 03:04:57.766340 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.739524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.27.109.2:40367)
0811 03:04:57.749739 (+ 10215us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:57.749754 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:57.754602 (+ 4848us) server_negotiation.cc:407] Connection header received
0811 03:04:57.754747 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:57.754754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:57.755005 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:57.755226 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:57.755937 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.757004 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.761807 (+ 4803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.764906 (+ 3099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.765058 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.765247 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:57.765838 (+ 591us) server_negotiation.cc:299] Negotiation successful
0811 03:04:57.766033 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10016,"thread_start_us":9892,"threads_started":1}
I20260811 03:04:57.768814 512 tablet_service.cc:1467] Processing CreateTablet for tablet 333c81031f9644559340d0e54888d4f1 (DEFAULT_TABLE table=test-table [id=eaae3e7362324087ae30ca177959374c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:57.768831 828 tablet_service.cc:1467] Processing CreateTablet for tablet 333c81031f9644559340d0e54888d4f1 (DEFAULT_TABLE table=test-table [id=eaae3e7362324087ae30ca177959374c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:57.770406 512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333c81031f9644559340d0e54888d4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:57.770427 828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333c81031f9644559340d0e54888d4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:57.771000 673 tablet_service.cc:1467] Processing CreateTablet for tablet 333c81031f9644559340d0e54888d4f1 (DEFAULT_TABLE table=test-table [id=eaae3e7362324087ae30ca177959374c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:04:57.772408 673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333c81031f9644559340d0e54888d4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:57.793413 921 tablet_bootstrap.cc:492] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Bootstrap starting.
I20260811 03:04:57.793740 922 tablet_bootstrap.cc:492] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Bootstrap starting.
I20260811 03:04:57.795001 920 tablet_bootstrap.cc:492] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Bootstrap starting.
I20260811 03:04:57.797608 922 tablet_bootstrap.cc:654] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:57.798802 922 log.cc:824] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:57.798977 921 tablet_bootstrap.cc:654] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:57.800356 921 log.cc:824] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:57.801628 920 tablet_bootstrap.cc:654] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:57.802301 922 tablet_bootstrap.cc:492] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: No bootstrap required, opened a new log
I20260811 03:04:57.802611 922 ts_tablet_manager.cc:1397] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 03:04:57.803083 920 log.cc:824] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:57.804064 921 tablet_bootstrap.cc:492] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: No bootstrap required, opened a new log
I20260811 03:04:57.804358 921 ts_tablet_manager.cc:1397] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260811 03:04:57.806615 920 tablet_bootstrap.cc:492] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: No bootstrap required, opened a new log
I20260811 03:04:57.807067 920 ts_tablet_manager.cc:1397] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.007s
I20260811 03:04:57.811887 922 raft_consensus.cc:348] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.812448 922 raft_consensus.cc:374] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:57.812562 922 raft_consensus.cc:729] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d258cdc26fec4dee863a1bde73222fae, State: Initialized, Role: FOLLOWER
I20260811 03:04:57.813323 922 consensus_queue.cc:260] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.814821 748 heartbeater.cc:503] Master 127.27.109.62:38595 was elected leader, sending a full tablet report...
I20260811 03:04:57.815490 922 ts_tablet_manager.cc:1428] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260811 03:04:57.816949 921 raft_consensus.cc:348] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.817488 921 raft_consensus.cc:374] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:57.817608 921 raft_consensus.cc:729] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312aca42d7be43619da0f6d9a3beec6c, State: Initialized, Role: FOLLOWER
I20260811 03:04:57.818317 921 consensus_queue.cc:260] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.818336 920 raft_consensus.cc:348] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.818744 920 raft_consensus.cc:374] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:57.818883 920 raft_consensus.cc:729] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46371fb0e0e494eba42fc42c25f5930, State: Initialized, Role: FOLLOWER
I20260811 03:04:57.819773 920 consensus_queue.cc:260] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.820802 921 ts_tablet_manager.cc:1428] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260811 03:04:57.821328 897 heartbeater.cc:503] Master 127.27.109.62:38595 was elected leader, sending a full tablet report...
I20260811 03:04:57.823232 920 ts_tablet_manager.cc:1428] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
W20260811 03:04:57.829224 749 tablet.cc:2369] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:04:57.901094 898 tablet.cc:2369] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:04:57.924055 926 raft_consensus.cc:482] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:04:57.924436 926 raft_consensus.cc:504] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.926281 926 leader_election.cc:288] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46371fb0e0e494eba42fc42c25f5930 (127.27.109.1:35375), 312aca42d7be43619da0f6d9a3beec6c (127.27.109.3:35727)
I20260811 03:04:57.933480 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.926927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:54947 (local address 127.27.109.3:35727)
0811 03:04:57.927338 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:57.927347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:57.927712 (+ 365us) server_negotiation.cc:407] Connection header received
0811 03:04:57.927969 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:57.927973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:57.928034 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:57.928133 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:57.928685 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.929440 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.931508 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.932313 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.932535 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.932621 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:57.933052 (+ 431us) server_negotiation.cc:299] Negotiation successful
0811 03:04:57.933193 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260811 03:04:57.933584 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.926819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34963 (local address 127.27.109.1:35375)
0811 03:04:57.927089 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:57.927095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:57.928101 (+ 1006us) server_negotiation.cc:407] Connection header received
0811 03:04:57.928294 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:57.928299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:57.928358 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:57.928456 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:04:57.929035 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.929764 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.931700 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.932497 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.932683 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.932767 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:57.933174 (+ 407us) server_negotiation.cc:299] Negotiation successful
0811 03:04:57.933316 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260811 03:04:57.933656 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.926598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:35727 (local address 127.27.109.2:54947)
0811 03:04:57.927590 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:57.927610 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:57.927802 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:57.928156 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:57.928162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:57.928189 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:57.928536 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.928546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.929575 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.929583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.931328 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.931342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.932557 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.932564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.932744 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.932831 (+ 87us) client_negotiation.cc:769] Sending connection context
0811 03:04:57.933022 (+ 191us) client_negotiation.cc:240] Negotiation successful
0811 03:04:57.933171 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":100,"thread_start_us":653,"threads_started":1}
I20260811 03:04:57.934198 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:57.926737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35375 (local address 127.27.109.2:34963)
0811 03:04:57.927975 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:57.927990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:57.928144 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:57.928487 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:57.928492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:57.928526 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:04:57.928876 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.928886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.929919 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.929924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.931520 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:04:57.931534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:04:57.932747 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:04:57.932754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:04:57.932999 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:57.933022 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 03:04:57.933152 (+ 130us) client_negotiation.cc:240] Negotiation successful
0811 03:04:57.933309 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":4,"thread_start_us":904,"threads_started":1}
I20260811 03:04:57.934849 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333c81031f9644559340d0e54888d4f1" candidate_uuid: "d258cdc26fec4dee863a1bde73222fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312aca42d7be43619da0f6d9a3beec6c" is_pre_election: true
I20260811 03:04:57.935530 848 raft_consensus.cc:2454] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d258cdc26fec4dee863a1bde73222fae in term 0.
I20260811 03:04:57.935411 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333c81031f9644559340d0e54888d4f1" candidate_uuid: "d258cdc26fec4dee863a1bde73222fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46371fb0e0e494eba42fc42c25f5930" is_pre_election: true
I20260811 03:04:57.935983 534 raft_consensus.cc:2454] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d258cdc26fec4dee863a1bde73222fae in term 0.
I20260811 03:04:57.936367 618 leader_election.cc:302] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 312aca42d7be43619da0f6d9a3beec6c, d258cdc26fec4dee863a1bde73222fae; no voters:
I20260811 03:04:57.936821 926 raft_consensus.cc:2781] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:04:57.936990 926 raft_consensus.cc:482] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:04:57.937099 926 raft_consensus.cc:3037] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:57.939462 926 raft_consensus.cc:504] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.940142 926 leader_election.cc:288] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [CANDIDATE]: Term 1 election: Requested vote from peers a46371fb0e0e494eba42fc42c25f5930 (127.27.109.1:35375), 312aca42d7be43619da0f6d9a3beec6c (127.27.109.3:35727)
I20260811 03:04:57.940694 534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333c81031f9644559340d0e54888d4f1" candidate_uuid: "d258cdc26fec4dee863a1bde73222fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46371fb0e0e494eba42fc42c25f5930"
I20260811 03:04:57.940749 848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333c81031f9644559340d0e54888d4f1" candidate_uuid: "d258cdc26fec4dee863a1bde73222fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312aca42d7be43619da0f6d9a3beec6c"
I20260811 03:04:57.940960 848 raft_consensus.cc:3037] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:57.940960 534 raft_consensus.cc:3037] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:57.944509 534 raft_consensus.cc:2454] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d258cdc26fec4dee863a1bde73222fae in term 1.
I20260811 03:04:57.944500 848 raft_consensus.cc:2454] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d258cdc26fec4dee863a1bde73222fae in term 1.
I20260811 03:04:57.945120 620 leader_election.cc:302] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a46371fb0e0e494eba42fc42c25f5930, d258cdc26fec4dee863a1bde73222fae; no voters:
I20260811 03:04:57.945511 926 raft_consensus.cc:2781] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:57.946120 926 raft_consensus.cc:686] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 1 LEADER]: Becoming Leader. State: Replica: d258cdc26fec4dee863a1bde73222fae, State: Running, Role: LEADER
I20260811 03:04:57.946969 926 consensus_queue.cc:237] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } }
I20260811 03:04:57.952196 372 catalog_manager.cc:5495] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae reported cstate change: term changed from 0 to 1, leader changed from <none> to d258cdc26fec4dee863a1bde73222fae (127.27.109.2). New cstate: current_term: 1 leader_uuid: "d258cdc26fec4dee863a1bde73222fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d258cdc26fec4dee863a1bde73222fae" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 40367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 } health_report { overall_health: UNKNOWN } } }
W20260811 03:04:57.989874 593 tablet.cc:2369] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:04:58.010986 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:58.004871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.27.109.2:40367)
0811 03:04:58.005121 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:58.005126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:58.005149 (+ 23us) server_negotiation.cc:407] Connection header received
0811 03:04:58.005339 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:58.005343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:58.005435 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:58.005587 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:04:58.006010 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:58.006905 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:58.008310 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:58.008744 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:58.008841 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:58.009526 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:04:58.010151 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:04:58.010279 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:58.010528 (+ 249us) server_negotiation.cc:299] Negotiation successful
0811 03:04:58.010647 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260811 03:04:58.021574 848 raft_consensus.cc:1261] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 1 FOLLOWER]: Refusing update from remote peer d258cdc26fec4dee863a1bde73222fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:58.021574 534 raft_consensus.cc:1261] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 1 FOLLOWER]: Refusing update from remote peer d258cdc26fec4dee863a1bde73222fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:04:58.022899 931 consensus_queue.cc:1035] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "312aca42d7be43619da0f6d9a3beec6c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:58.023316 926 consensus_queue.cc:1035] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46371fb0e0e494eba42fc42c25f5930" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:04:58.039459 938 mvcc.cc:205] Tried to move back new op lower bound from 7317166071883325440 to 7317166071595192320. Current Snapshot: MvccSnapshot[applied={T|T < 7317166071883325440}]
I20260811 03:04:58.053098 373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:58.053411 373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:04:58.057315 373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9574efc4787e494587d60e77d55dc8ba: Sending DeleteTablet for 3 replicas of tablet 333c81031f9644559340d0e54888d4f1
I20260811 03:04:58.058866 828 tablet_service.cc:1514] Processing DeleteTablet for tablet 333c81031f9644559340d0e54888d4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:58412
I20260811 03:04:58.059163 673 tablet_service.cc:1514] Processing DeleteTablet for tablet 333c81031f9644559340d0e54888d4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:38724
I20260811 03:04:58.059242 512 tablet_service.cc:1514] Processing DeleteTablet for tablet 333c81031f9644559340d0e54888d4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:04:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:49658
I20260811 03:04:58.061352 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 445
I20260811 03:04:58.068809 949 tablet_replica.cc:330] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: stopping tablet replica
I20260811 03:04:58.069458 949 raft_consensus.cc:2229] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:58.070118 949 raft_consensus.cc:2258] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:58.075418 949 ts_tablet_manager.cc:1905] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:58.082647 948 tablet_replica.cc:330] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: stopping tablet replica
I20260811 03:04:58.083213 948 raft_consensus.cc:2229] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:04:58.084136 948 raft_consensus.cc:2258] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 11 03:04:58 dist-test-slave-8tdl krb5kdc[324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:04:58.088997 947 tablet_replica.cc:330] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: stopping tablet replica
I20260811 03:04:58.089620 947 raft_consensus.cc:2229] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:04:58.090286 947 raft_consensus.cc:2258] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:04:58.092924 948 ts_tablet_manager.cc:1905] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:58.098045 949 ts_tablet_manager.cc:1918] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:04:58.098251 949 log.cc:1196] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/333c81031f9644559340d0e54888d4f1
I20260811 03:04:58.100399 947 ts_tablet_manager.cc:1905] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:04:58.104310 948 ts_tablet_manager.cc:1918] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:04:58.104530 948 log.cc:1196] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/333c81031f9644559340d0e54888d4f1
I20260811 03:04:58.105245 948 ts_tablet_manager.cc:1939] T 333c81031f9644559340d0e54888d4f1 P d258cdc26fec4dee863a1bde73222fae: Deleting consensus metadata
I20260811 03:04:58.107443 359 catalog_manager.cc:4841] TS d258cdc26fec4dee863a1bde73222fae (127.27.109.2:40367): tablet 333c81031f9644559340d0e54888d4f1 (table test-table [id=eaae3e7362324087ae30ca177959374c]) successfully deleted
I20260811 03:04:58.107766 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:58.063125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.27.109.1:35375)
0811 03:04:58.063376 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:58.063385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:58.075924 (+ 12539us) server_negotiation.cc:407] Connection header received
0811 03:04:58.076249 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:58.076258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:58.076356 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:58.076434 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:58.078262 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:58.079670 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:58.081960 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:58.082664 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:58.082804 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:58.093803 (+ 10999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:58.093828 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:58.093849 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:58.093915 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:58.098462 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:58.100728 (+ 2266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:58.100753 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:58.100759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:58.100976 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:58.105222 (+ 4246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:58.105231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:58.105236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:58.105877 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:58.106214 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:58.106980 (+ 766us) server_negotiation.cc:299] Negotiation successful
0811 03:04:58.107280 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 03:04:58.108031 949 ts_tablet_manager.cc:1939] T 333c81031f9644559340d0e54888d4f1 P a46371fb0e0e494eba42fc42c25f5930: Deleting consensus metadata
I20260811 03:04:58.110504 359 catalog_manager.cc:4841] TS a46371fb0e0e494eba42fc42c25f5930 (127.27.109.1:35375): tablet 333c81031f9644559340d0e54888d4f1 (table test-table [id=eaae3e7362324087ae30ca177959374c]) successfully deleted
I20260811 03:04:58.111575 587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:58.113809 947 ts_tablet_manager.cc:1918] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:04:58.114040 947 log.cc:1196] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/333c81031f9644559340d0e54888d4f1
I20260811 03:04:58.114663 947 ts_tablet_manager.cc:1939] T 333c81031f9644559340d0e54888d4f1 P 312aca42d7be43619da0f6d9a3beec6c: Deleting consensus metadata
I20260811 03:04:58.116962 357 catalog_manager.cc:4841] TS 312aca42d7be43619da0f6d9a3beec6c (127.27.109.3:35727): tablet 333c81031f9644559340d0e54888d4f1 (table test-table [id=eaae3e7362324087ae30ca177959374c]) successfully deleted
I20260811 03:04:58.205273 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 445
I20260811 03:04:58.228547 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 600
Aug 11 03:04:58 dist-test-slave-8tdl krb5kdc[324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:04:58.246654 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:58.229424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.27.109.2:40367)
0811 03:04:58.229610 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:58.229618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:58.229722 (+ 104us) server_negotiation.cc:407] Connection header received
0811 03:04:58.229943 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:58.229954 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:58.230027 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:58.230145 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:58.231514 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:58.232371 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:58.233670 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:58.234151 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:58.234298 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:58.239327 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:58.239349 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:58.239366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:58.239416 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:58.243338 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:58.244149 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:58.244157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:58.244160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:58.244319 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:58.244732 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:58.244737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:58.244745 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:58.245235 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:58.245617 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:58.246034 (+ 417us) server_negotiation.cc:299] Negotiation successful
0811 03:04:58.246252 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:04:58.247439 742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:58.370342 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 600
I20260811 03:04:58.392462 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 755
Aug 11 03:04:58 dist-test-slave-8tdl krb5kdc[324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:04:58.409056 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:58.393366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.27.109.3:35727)
0811 03:04:58.393541 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:58.393545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:58.393565 (+ 20us) server_negotiation.cc:407] Connection header received
0811 03:04:58.393737 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:58.393742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:58.393807 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:58.393898 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:58.395338 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:58.396172 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:58.397542 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:04:58.397942 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:04:58.398043 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:04:58.402665 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:58.402753 (+ 88us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:58.402785 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:04:58.402846 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:58.406122 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:58.406970 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:58.406979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:58.406981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:58.407165 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:04:58.407566 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:04:58.407570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:04:58.407572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:04:58.407968 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:58.408204 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:58.408593 (+ 389us) server_negotiation.cc:299] Negotiation successful
0811 03:04:58.408776 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 03:04:58.409754 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:58.505183 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 755
I20260811 03:04:58.528069 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 338
I20260811 03:04:58.529382 405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:04:58.651393 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 338
2026-08-11T03:04:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4853 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-11T03:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:04:58Z Disabled control of system clock
I20260811 03:04:58.736882 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38071
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 03:04:59.103502 958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:59.103829 958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:59.103997 958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:59.111588 958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:04:59.111712 958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:59.111785 958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:59.111877 958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:04:59.111958 958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:04:59.122323 958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:59.123775 958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:59.125674 958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:59.139159 966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:59.139344 965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:59.139326 969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:59.140228 958 server_base.cc:1034] running on GCE node
I20260811 03:04:59.141194 958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:59.142935 958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:59.144219 958 hybrid_clock.cc:648] HybridClock initialized: now 1786417499144153 us; error 61 us; skew 500 ppm
I20260811 03:04:59.144708 958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:04:59.147362 958 webserver.cc:460] Webserver started at http://127.27.109.62:35873/ using document root <none> and password file <none>
I20260811 03:04:59.148161 958 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:59.148303 958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:59.148685 958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:59.152016 958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "122cfacd7f61481b9e50be759713038e"
format_stamp: "Formatted at 2026-08-11 03:04:59 on dist-test-slave-8tdl"
server_key: "8250397ac099d2f3914f7bf85898921f"
server_key_iv: "947389df361bfb7f34688797cbe889d7"
server_key_version: "encryptionkey@0"
I20260811 03:04:59.153049 958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "122cfacd7f61481b9e50be759713038e"
format_stamp: "Formatted at 2026-08-11 03:04:59 on dist-test-slave-8tdl"
server_key: "8250397ac099d2f3914f7bf85898921f"
server_key_iv: "947389df361bfb7f34688797cbe889d7"
server_key_version: "encryptionkey@0"
I20260811 03:04:59.159471 958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260811 03:04:59.163887 976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:59.165465 958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:04:59.165729 958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "122cfacd7f61481b9e50be759713038e"
format_stamp: "Formatted at 2026-08-11 03:04:59 on dist-test-slave-8tdl"
server_key: "8250397ac099d2f3914f7bf85898921f"
server_key_iv: "947389df361bfb7f34688797cbe889d7"
server_key_version: "encryptionkey@0"
I20260811 03:04:59.165993 958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:59.172735 958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:59.173661 958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:59.174003 958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:59.195338 958 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:38071
I20260811 03:04:59.195374 1032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:38071 every 8 connection(s)
I20260811 03:04:59.196944 958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:04:59.201231 1033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:04:59.203203 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 958
I20260811 03:04:59.203428 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:04:59.204146 28084 external_mini_cluster.cc:1428] Setting key a87a1350eab3f8d9bb6551d272b2b835
I20260811 03:04:59.215204 1037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:59.206428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.27.109.62:38071)
0811 03:04:59.207718 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:59.207745 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:59.207807 (+ 62us) server_negotiation.cc:407] Connection header received
0811 03:04:59.210345 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:59.210381 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:59.211194 (+ 813us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:59.211742 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:59.212931 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:59.212941 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:59.212966 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:04:59.212990 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:59.213076 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:04:59.213098 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:59.213353 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:59.213500 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 03:04:59.213823 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":451,"threads_started":1}
I20260811 03:04:59.216075 1033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e: Bootstrap starting.
I20260811 03:04:59.219306 1033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e: Neither blocks nor log segments found. Creating new log.
I20260811 03:04:59.220402 1033 log.cc:824] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e: Log is configured to *not* fsync() on all Append() calls
I20260811 03:04:59.224241 1033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e: No bootstrap required, opened a new log
I20260811 03:04:59.233178 1033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122cfacd7f61481b9e50be759713038e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38071 } }
I20260811 03:04:59.233703 1033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:04:59.233829 1033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 122cfacd7f61481b9e50be759713038e, State: Initialized, Role: FOLLOWER
I20260811 03:04:59.234520 1033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122cfacd7f61481b9e50be759713038e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38071 } }
I20260811 03:04:59.234901 1033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:04:59.235075 1033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:04:59.235273 1033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:04:59.238515 1033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122cfacd7f61481b9e50be759713038e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38071 } }
I20260811 03:04:59.239282 1033 leader_election.cc:302] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 122cfacd7f61481b9e50be759713038e; no voters:
I20260811 03:04:59.240070 1033 leader_election.cc:288] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:04:59.240231 1039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:04:59.241052 1039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [term 1 LEADER]: Becoming Leader. State: Replica: 122cfacd7f61481b9e50be759713038e, State: Running, Role: LEADER
I20260811 03:04:59.241671 1039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122cfacd7f61481b9e50be759713038e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38071 } }
I20260811 03:04:59.242568 1033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:04:59.245185 1042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 122cfacd7f61481b9e50be759713038e. Latest consensus state: current_term: 1 leader_uuid: "122cfacd7f61481b9e50be759713038e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122cfacd7f61481b9e50be759713038e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38071 } } }
I20260811 03:04:59.245043 1040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "122cfacd7f61481b9e50be759713038e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "122cfacd7f61481b9e50be759713038e" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38071 } } }
I20260811 03:04:59.245832 1042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:59.245833 1040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e [sys.catalog]: This master's current role is: LEADER
I20260811 03:04:59.246838 1048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:04:59.252485 1048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:04:59.261950 1048 catalog_manager.cc:1347] Generated new cluster ID: 3aa896bac1ef4207a5b928c94b8cad91
I20260811 03:04:59.262092 1048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:04:59.292062 1048 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:04:59.292544 1048 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:04:59.302647 1048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e: Generated new TSK 0
I20260811 03:04:59.303619 1048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:04:59.318743 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38071
--builtin_ntp_servers=127.27.109.20:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 03:04:59.699267 1063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:04:59.699604 1063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:04:59.699762 1063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:04:59.708106 1063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:04:59.708242 1063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:04:59.708431 1063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:04:59.719177 1063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:04:59.721196 1063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:04:59.723671 1063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:04:59.741174 1069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:59.741092 1070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:04:59.741110 1073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:04:59.742223 1063 server_base.cc:1034] running on GCE node
I20260811 03:04:59.743161 1063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:04:59.744393 1063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:04:59.745685 1063 hybrid_clock.cc:648] HybridClock initialized: now 1786417499745636 us; error 63 us; skew 500 ppm
I20260811 03:04:59.746078 1063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:04:59.749693 1063 webserver.cc:460] Webserver started at http://127.27.109.1:43989/ using document root <none> and password file <none>
I20260811 03:04:59.750519 1063 fs_manager.cc:359] Metadata directory not provided
I20260811 03:04:59.750662 1063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:04:59.751072 1063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:04:59.754266 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "32b898cf52eb46cc96db13f37367e7fe"
format_stamp: "Formatted at 2026-08-11 03:04:59 on dist-test-slave-8tdl"
server_key: "618d1660db84181eff7d4b679c3e882b"
server_key_iv: "c33548a1c5e964ccf34b1b7921da7274"
server_key_version: "encryptionkey@0"
I20260811 03:04:59.755213 1063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "32b898cf52eb46cc96db13f37367e7fe"
format_stamp: "Formatted at 2026-08-11 03:04:59 on dist-test-slave-8tdl"
server_key: "618d1660db84181eff7d4b679c3e882b"
server_key_iv: "c33548a1c5e964ccf34b1b7921da7274"
server_key_version: "encryptionkey@0"
I20260811 03:04:59.760895 1063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 03:04:59.764909 1080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:59.766582 1063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:04:59.766845 1063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "32b898cf52eb46cc96db13f37367e7fe"
format_stamp: "Formatted at 2026-08-11 03:04:59 on dist-test-slave-8tdl"
server_key: "618d1660db84181eff7d4b679c3e882b"
server_key_iv: "c33548a1c5e964ccf34b1b7921da7274"
server_key_version: "encryptionkey@0"
I20260811 03:04:59.767146 1063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:04:59.773921 1063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:04:59.775002 1063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:04:59.775336 1063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:04:59.776500 1063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:04:59.778242 1063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:04:59.778380 1063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:59.778537 1063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:04:59.778632 1063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:04:59.814894 1063 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:32881
I20260811 03:04:59.814945 1205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:32881 every 8 connection(s)
I20260811 03:04:59.816329 1063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:04:59.819276 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 1063
I20260811 03:04:59.819533 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:04:59.820288 28084 external_mini_cluster.cc:1428] Setting key 4ba73c4af1ae3234d557614db614a201
I20260811 03:04:59.823483 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38071
--builtin_ntp_servers=127.27.109.20:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 03:04:59.825788 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:59.819173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:47821 (local address 127.27.109.62:38071)
0811 03:04:59.819726 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0811 03:04:59.819733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:04:59.821177 (+ 1444us) server_negotiation.cc:407] Connection header received
0811 03:04:59.823190 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:04:59.823198 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:04:59.823274 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:04:59.823395 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:04:59.824885 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:04:59.824895 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:04:59.824902 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:04:59.824928 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:04:59.824992 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:04:59.825012 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:04:59.825166 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0811 03:04:59.825311 (+ 145us) server_negotiation.cc:299] Negotiation successful
0811 03:04:59.825383 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
I20260811 03:04:59.826633 1209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:04:59.819838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38071 (local address 127.27.109.1:47821)
0811 03:04:59.821001 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0811 03:04:59.821047 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 03:04:59.822913 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:04:59.823680 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:04:59.823692 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:04:59.823816 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:04:59.824531 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:04:59.824533 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:04:59.824608 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:04:59.824700 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:04:59.825116 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:04:59.825124 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:04:59.825129 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 03:04:59.825346 (+ 217us) client_negotiation.cc:240] Negotiation successful
0811 03:04:59.825587 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":398,"threads_started":1}
I20260811 03:04:59.828397 1206 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38071
I20260811 03:04:59.828719 1206 heartbeater.cc:464] Registering TS with master...
I20260811 03:04:59.829330 1206 heartbeater.cc:511] Master 127.27.109.62:38071 requested a full tablet report, sending...
I20260811 03:04:59.831537 995 ts_manager.cc:194] Registered new tserver with Master: 32b898cf52eb46cc96db13f37367e7fe (127.27.109.1:32881)
W20260811 03:05:00.197147 1210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:00.197453 1210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:00.197613 1210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:00.205641 1210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:00.205760 1210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:00.205943 1210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:00.217609 1210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:00.219201 1210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:00.221304 1210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:00.236327 1215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:00.236222 1216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:00.236287 1219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:00.237044 1210 server_base.cc:1034] running on GCE node
I20260811 03:05:00.237757 1210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:00.238848 1210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:00.240072 1210 hybrid_clock.cc:648] HybridClock initialized: now 1786417500240017 us; error 44 us; skew 500 ppm
I20260811 03:05:00.240518 1210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:00.243407 1210 webserver.cc:460] Webserver started at http://127.27.109.2:38649/ using document root <none> and password file <none>
I20260811 03:05:00.244477 1210 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:00.244594 1210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:00.245011 1210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:00.248620 1210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "173c1a53b92a4edc95ddf8f5d83e8e62"
format_stamp: "Formatted at 2026-08-11 03:05:00 on dist-test-slave-8tdl"
server_key: "f8da9a85d07b3243b2a5a226121ba545"
server_key_iv: "d1846821d8d2bd945675856473e9d815"
server_key_version: "encryptionkey@0"
I20260811 03:05:00.249558 1210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "173c1a53b92a4edc95ddf8f5d83e8e62"
format_stamp: "Formatted at 2026-08-11 03:05:00 on dist-test-slave-8tdl"
server_key: "f8da9a85d07b3243b2a5a226121ba545"
server_key_iv: "d1846821d8d2bd945675856473e9d815"
server_key_version: "encryptionkey@0"
I20260811 03:05:00.255716 1210 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260811 03:05:00.259948 1225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:00.261647 1210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:05:00.261827 1210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "173c1a53b92a4edc95ddf8f5d83e8e62"
format_stamp: "Formatted at 2026-08-11 03:05:00 on dist-test-slave-8tdl"
server_key: "f8da9a85d07b3243b2a5a226121ba545"
server_key_iv: "d1846821d8d2bd945675856473e9d815"
server_key_version: "encryptionkey@0"
I20260811 03:05:00.262070 1210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:00.268788 1210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:00.269754 1210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:00.270026 1210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:00.271196 1210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:00.272998 1210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:00.273100 1210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:00.273198 1210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:00.273286 1210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:00.312685 1210 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:35565
I20260811 03:05:00.312741 1357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:35565 every 8 connection(s)
I20260811 03:05:00.314244 1210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:00.322508 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 1210
I20260811 03:05:00.322773 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:05:00.323269 28084 external_mini_cluster.cc:1428] Setting key d2f0b0affa511869988f880c38318f6f
I20260811 03:05:00.323660 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.317182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:37877 (local address 127.27.109.62:38071)
0811 03:05:00.317408 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:00.317415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:00.319069 (+ 1654us) server_negotiation.cc:407] Connection header received
0811 03:05:00.321041 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:00.321046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:00.321109 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:00.321206 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:00.322775 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:00.322781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:00.322787 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:00.322816 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:00.322871 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:00.322892 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:00.323041 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:00.323261 (+ 220us) server_negotiation.cc:299] Negotiation successful
0811 03:05:00.323347 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 03:05:00.324980 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.317761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38071 (local address 127.27.109.2:37877)
0811 03:05:00.318885 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:00.318929 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:00.320735 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:00.321443 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:00.321452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:00.321586 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:00.322367 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:00.322370 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:00.322412 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:00.322510 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:00.323005 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:00.323019 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:00.323028 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 03:05:00.323357 (+ 329us) client_negotiation.cc:240] Negotiation successful
0811 03:05:00.323672 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":364,"threads_started":1}
I20260811 03:05:00.326666 1358 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38071
I20260811 03:05:00.327078 1358 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:00.327122 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38071
--builtin_ntp_servers=127.27.109.20:36755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 03:05:00.327903 1358 heartbeater.cc:511] Master 127.27.109.62:38071 requested a full tablet report, sending...
I20260811 03:05:00.329931 995 ts_manager.cc:194] Registered new tserver with Master: 173c1a53b92a4edc95ddf8f5d83e8e62 (127.27.109.2:35565)
W20260811 03:05:00.702520 1362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:00.702975 1362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:00.703155 1362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:00.710994 1362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:00.711158 1362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:00.711349 1362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:00.721675 1362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:00.723443 1362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:00.725986 1362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:00.742291 1367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:00.742340 1369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:00.742450 1372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:00.743410 1362 server_base.cc:1034] running on GCE node
I20260811 03:05:00.744143 1362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:00.745291 1362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:00.746644 1362 hybrid_clock.cc:648] HybridClock initialized: now 1786417500746581 us; error 47 us; skew 500 ppm
I20260811 03:05:00.747140 1362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:00.750316 1362 webserver.cc:460] Webserver started at http://127.27.109.3:36487/ using document root <none> and password file <none>
I20260811 03:05:00.751230 1362 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:00.751360 1362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:00.751744 1362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:00.754997 1362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "19e412d4b89c4cef9b820d349f7ac5ec"
format_stamp: "Formatted at 2026-08-11 03:05:00 on dist-test-slave-8tdl"
server_key: "8fda8f5abe331dbcf1e62ffedf43ada6"
server_key_iv: "76816063b5d4c37c740b2be646efc901"
server_key_version: "encryptionkey@0"
I20260811 03:05:00.755910 1362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "19e412d4b89c4cef9b820d349f7ac5ec"
format_stamp: "Formatted at 2026-08-11 03:05:00 on dist-test-slave-8tdl"
server_key: "8fda8f5abe331dbcf1e62ffedf43ada6"
server_key_iv: "76816063b5d4c37c740b2be646efc901"
server_key_version: "encryptionkey@0"
I20260811 03:05:00.761946 1362 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:05:00.766317 1379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:00.768133 1362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:00.768347 1362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "19e412d4b89c4cef9b820d349f7ac5ec"
format_stamp: "Formatted at 2026-08-11 03:05:00 on dist-test-slave-8tdl"
server_key: "8fda8f5abe331dbcf1e62ffedf43ada6"
server_key_iv: "76816063b5d4c37c740b2be646efc901"
server_key_version: "encryptionkey@0"
I20260811 03:05:00.768579 1362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:00.775545 1362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:00.776501 1362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:00.776811 1362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:00.777845 1362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:00.779850 1362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:00.779983 1362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:00.780125 1362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:00.780222 1362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:00.816922 1362 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:37741
I20260811 03:05:00.817039 1508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:37741 every 8 connection(s)
I20260811 03:05:00.818413 1362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:00.825830 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 1362
I20260811 03:05:00.826059 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:05:00.826665 28084 external_mini_cluster.cc:1428] Setting key a5f0a57094193796dbcc05d4f569878c
I20260811 03:05:00.828480 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.821311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:40475 (local address 127.27.109.62:38071)
0811 03:05:00.821505 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:00.821509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:00.823246 (+ 1737us) server_negotiation.cc:407] Connection header received
0811 03:05:00.825396 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:00.825403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:00.825485 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:00.825624 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:00.827619 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:00.827627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:00.827632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:00.827661 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:00.827711 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:00.827726 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:00.827839 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:00.828094 (+ 255us) server_negotiation.cc:299] Negotiation successful
0811 03:05:00.828165 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 03:05:00.829782 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.821867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38071 (local address 127.27.109.3:40475)
0811 03:05:00.823026 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:00.823083 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:00.825009 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:00.825970 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:00.825985 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:00.826204 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:00.827163 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:00.827167 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:00.827229 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:00.827372 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:00.827837 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:00.827849 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:00.827857 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 03:05:00.828199 (+ 342us) client_negotiation.cc:240] Negotiation successful
0811 03:05:00.828519 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":345,"threads_started":1}
I20260811 03:05:00.831357 1510 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38071
I20260811 03:05:00.831722 1510 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:00.832386 1510 heartbeater.cc:511] Master 127.27.109.62:38071 requested a full tablet report, sending...
I20260811 03:05:00.833922 995 ts_manager.cc:194] Registered new tserver with Master: 19e412d4b89c4cef9b820d349f7ac5ec (127.27.109.3:37741)
I20260811 03:05:00.834663 1206 heartbeater.cc:503] Master 127.27.109.62:38071 was elected leader, sending a full tablet report...
I20260811 03:05:00.843879 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:05:00.852306 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.849292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.27.109.62:38071)
0811 03:05:00.849534 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:00.849574 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:00.850005 (+ 431us) server_negotiation.cc:407] Connection header received
0811 03:05:00.850270 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:00.850276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:00.850340 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:00.850450 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:00.851598 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:00.851604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:00.851609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:00.851633 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:00.851671 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:00.851684 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:00.851827 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:00.851920 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 03:05:00.851995 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 03:05:00.857709 995 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:05:00.866851 995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:05:00.890973 1523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.883193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.27.109.3:37741)
0811 03:05:00.884683 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:00.884691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:00.885688 (+ 997us) server_negotiation.cc:407] Connection header received
0811 03:05:00.885962 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:00.885975 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:00.886276 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:00.886416 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:00.889684 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:00.889696 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:00.889711 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:00.889736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:00.889818 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:00.889843 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:00.890029 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:00.890368 (+ 339us) server_negotiation.cc:299] Negotiation successful
0811 03:05:00.890503 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":437,"threads_started":1}
I20260811 03:05:00.891027 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.883394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:35565 (local address 127.0.0.1:56198)
0811 03:05:00.886213 (+ 2819us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:00.886233 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:00.886348 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:00.888018 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:00.888023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:00.888048 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:00.889681 (+ 1633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:00.889684 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:00.889702 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:00.889751 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:00.890314 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:00.890319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:00.890323 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 03:05:00.890474 (+ 151us) client_negotiation.cc:240] Negotiation successful
0811 03:05:00.890570 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2678,"thread_start_us":1846,"threads_started":1}
I20260811 03:05:00.891237 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.884011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.27.109.2:35565)
0811 03:05:00.884591 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:00.884613 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:00.886488 (+ 1875us) server_negotiation.cc:407] Connection header received
0811 03:05:00.886575 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:00.886583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:00.886830 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:00.887008 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:00.889962 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:00.889973 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:00.889988 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:00.890024 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:00.890091 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:00.890118 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:00.890260 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:00.890701 (+ 441us) server_negotiation.cc:299] Negotiation successful
0811 03:05:00.890830 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":274,"threads_started":1}
I20260811 03:05:00.892488 1525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.883138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:37741 (local address 127.0.0.1:35618)
0811 03:05:00.885507 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:00.885549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:00.885750 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:00.887831 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:00.887839 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:00.887951 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:00.889334 (+ 1383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:00.889338 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:00.889374 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:00.889476 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:00.890111 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:00.890124 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:00.890132 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 03:05:00.891937 (+ 1805us) client_negotiation.cc:240] Negotiation successful
0811 03:05:00.892009 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2143,"thread_start_us":2721,"threads_started":1}
I20260811 03:05:00.893823 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.886727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.27.109.1:32881)
0811 03:05:00.888748 (+ 2021us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:00.888758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:00.889326 (+ 568us) server_negotiation.cc:407] Connection header received
0811 03:05:00.889416 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:00.889423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:00.889704 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:00.889866 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:00.892927 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:00.892937 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:00.892952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:00.892976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:00.893059 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:00.893086 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:00.893286 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:00.893359 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 03:05:00.893461 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1814,"thread_start_us":1603,"threads_started":1}
I20260811 03:05:00.893836 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:00.886360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:32881 (local address 127.0.0.1:51556)
0811 03:05:00.889137 (+ 2777us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:00.889157 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:00.889276 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:00.889863 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:00.889868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:00.889894 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:00.891992 (+ 2098us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:00.891996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:00.892015 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:00.892086 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:00.893196 (+ 1110us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:00.893201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:00.893206 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 03:05:00.893345 (+ 139us) client_negotiation.cc:240] Negotiation successful
0811 03:05:00.893406 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2638,"thread_start_us":2464,"threads_started":1}
I20260811 03:05:00.895707 1285 tablet_service.cc:1467] Processing CreateTablet for tablet e62a7bd1296d4f1385fa1ef28a7d2f52 (DEFAULT_TABLE table=test-table [id=e7916a18b41b4897941d090068b91cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:00.896521 1435 tablet_service.cc:1467] Processing CreateTablet for tablet e62a7bd1296d4f1385fa1ef28a7d2f52 (DEFAULT_TABLE table=test-table [id=e7916a18b41b4897941d090068b91cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:00.897457 1285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62a7bd1296d4f1385fa1ef28a7d2f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:00.897799 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62a7bd1296d4f1385fa1ef28a7d2f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:00.898216 1133 tablet_service.cc:1467] Processing CreateTablet for tablet e62a7bd1296d4f1385fa1ef28a7d2f52 (DEFAULT_TABLE table=test-table [id=e7916a18b41b4897941d090068b91cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:00.899809 1133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62a7bd1296d4f1385fa1ef28a7d2f52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:00.911559 1530 tablet_bootstrap.cc:492] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Bootstrap starting.
I20260811 03:05:00.914053 1531 tablet_bootstrap.cc:492] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Bootstrap starting.
I20260811 03:05:00.916523 1530 tablet_bootstrap.cc:654] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:00.918711 1530 log.cc:824] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:00.919334 1531 tablet_bootstrap.cc:654] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:00.920994 1531 log.cc:824] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:00.922029 1532 tablet_bootstrap.cc:492] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Bootstrap starting.
I20260811 03:05:00.922946 1530 tablet_bootstrap.cc:492] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: No bootstrap required, opened a new log
I20260811 03:05:00.923348 1530 ts_tablet_manager.cc:1397] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260811 03:05:00.924690 1531 tablet_bootstrap.cc:492] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: No bootstrap required, opened a new log
I20260811 03:05:00.924983 1531 ts_tablet_manager.cc:1397] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260811 03:05:00.926201 1532 tablet_bootstrap.cc:654] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:00.927615 1532 log.cc:824] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:00.931493 1532 tablet_bootstrap.cc:492] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: No bootstrap required, opened a new log
I20260811 03:05:00.931845 1532 ts_tablet_manager.cc:1397] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260811 03:05:00.933840 1531 raft_consensus.cc:348] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:00.934283 1531 raft_consensus.cc:374] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:00.934232 1530 raft_consensus.cc:348] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:00.934427 1531 raft_consensus.cc:729] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173c1a53b92a4edc95ddf8f5d83e8e62, State: Initialized, Role: FOLLOWER
I20260811 03:05:00.934746 1530 raft_consensus.cc:374] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:00.934898 1530 raft_consensus.cc:729] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e412d4b89c4cef9b820d349f7ac5ec, State: Initialized, Role: FOLLOWER
I20260811 03:05:00.935613 1531 consensus_queue.cc:260] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:00.935720 1530 consensus_queue.cc:260] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:00.939541 1358 heartbeater.cc:503] Master 127.27.109.62:38071 was elected leader, sending a full tablet report...
I20260811 03:05:00.941800 1531 ts_tablet_manager.cc:1428] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260811 03:05:00.943087 1510 heartbeater.cc:503] Master 127.27.109.62:38071 was elected leader, sending a full tablet report...
I20260811 03:05:00.944257 1530 ts_tablet_manager.cc:1428] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Time spent starting tablet: real 0.021s user 0.006s sys 0.013s
I20260811 03:05:00.946228 1532 raft_consensus.cc:348] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:00.946907 1532 raft_consensus.cc:374] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:00.947046 1532 raft_consensus.cc:729] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b898cf52eb46cc96db13f37367e7fe, State: Initialized, Role: FOLLOWER
I20260811 03:05:00.947813 1532 consensus_queue.cc:260] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:00.949918 1532 ts_tablet_manager.cc:1428] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Time spent starting tablet: real 0.018s user 0.015s sys 0.001s
I20260811 03:05:01.008797 1541 raft_consensus.cc:482] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:01.009285 1541 raft_consensus.cc:504] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:01.011415 1541 leader_election.cc:288] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19e412d4b89c4cef9b820d349f7ac5ec (127.27.109.3:37741), 173c1a53b92a4edc95ddf8f5d83e8e62 (127.27.109.2:35565)
I20260811 03:05:01.015774 1523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:01.012164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44605 (local address 127.27.109.3:37741)
0811 03:05:01.012352 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:01.012357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:01.013145 (+ 788us) server_negotiation.cc:407] Connection header received
0811 03:05:01.013388 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:01.013394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:01.013451 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:01.013562 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:01.014934 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:01.014944 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:01.014951 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:01.014979 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:01.015025 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:01.015044 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:01.015210 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:01.015374 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 03:05:01.015460 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:05:01.016072 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:01.012023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:43039 (local address 127.27.109.2:35565)
0811 03:05:01.012260 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:01.012267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:01.013651 (+ 1384us) server_negotiation.cc:407] Connection header received
0811 03:05:01.013853 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:01.013859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:01.013917 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:01.014037 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:01.015262 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:01.015269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:01.015276 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:01.015303 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:01.015352 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:01.015374 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:01.015494 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:01.015636 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 03:05:01.015714 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 03:05:01.016122 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:01.012018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:37741 (local address 127.27.109.1:44605)
0811 03:05:01.012979 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:01.013002 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:01.013205 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:01.013595 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:01.013602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:01.013629 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:01.014515 (+ 886us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:01.014518 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:01.014537 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:01.014580 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:01.015226 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:01.015231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:01.015236 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 03:05:01.015398 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 03:05:01.015506 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"spinlock_wait_cycles":156032,"thread_start_us":1156,"threads_started":1}
I20260811 03:05:01.016247 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:01.012060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:35565 (local address 127.27.109.1:43039)
0811 03:05:01.013525 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:01.013541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:01.013704 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:01.014060 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:01.014065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:01.014091 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:01.015006 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:01.015010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:01.015027 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:01.015074 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:01.015483 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:01.015488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:01.015493 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 03:05:01.015661 (+ 168us) client_negotiation.cc:240] Negotiation successful
0811 03:05:01.015716 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"thread_start_us":595,"threads_started":1}
I20260811 03:05:01.017266 1307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62a7bd1296d4f1385fa1ef28a7d2f52" candidate_uuid: "32b898cf52eb46cc96db13f37367e7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" is_pre_election: true
I20260811 03:05:01.017267 1459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62a7bd1296d4f1385fa1ef28a7d2f52" candidate_uuid: "32b898cf52eb46cc96db13f37367e7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" is_pre_election: true
I20260811 03:05:01.017843 1459 raft_consensus.cc:2454] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32b898cf52eb46cc96db13f37367e7fe in term 0.
I20260811 03:05:01.017844 1307 raft_consensus.cc:2454] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32b898cf52eb46cc96db13f37367e7fe in term 0.
I20260811 03:05:01.018615 1081 leader_election.cc:302] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 173c1a53b92a4edc95ddf8f5d83e8e62, 19e412d4b89c4cef9b820d349f7ac5ec, 32b898cf52eb46cc96db13f37367e7fe; no voters:
I20260811 03:05:01.019181 1541 raft_consensus.cc:2781] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:01.019348 1541 raft_consensus.cc:482] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:01.019480 1541 raft_consensus.cc:3037] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:01.021881 1541 raft_consensus.cc:504] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:01.022544 1541 leader_election.cc:288] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [CANDIDATE]: Term 1 election: Requested vote from peers 19e412d4b89c4cef9b820d349f7ac5ec (127.27.109.3:37741), 173c1a53b92a4edc95ddf8f5d83e8e62 (127.27.109.2:35565)
I20260811 03:05:01.023226 1459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62a7bd1296d4f1385fa1ef28a7d2f52" candidate_uuid: "32b898cf52eb46cc96db13f37367e7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19e412d4b89c4cef9b820d349f7ac5ec"
I20260811 03:05:01.023260 1307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62a7bd1296d4f1385fa1ef28a7d2f52" candidate_uuid: "32b898cf52eb46cc96db13f37367e7fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62"
I20260811 03:05:01.023483 1307 raft_consensus.cc:3037] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:01.023532 1459 raft_consensus.cc:3037] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:01.027053 1459 raft_consensus.cc:2454] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32b898cf52eb46cc96db13f37367e7fe in term 1.
I20260811 03:05:01.027047 1307 raft_consensus.cc:2454] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32b898cf52eb46cc96db13f37367e7fe in term 1.
I20260811 03:05:01.027658 1082 leader_election.cc:302] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 173c1a53b92a4edc95ddf8f5d83e8e62, 19e412d4b89c4cef9b820d349f7ac5ec, 32b898cf52eb46cc96db13f37367e7fe; no voters:
I20260811 03:05:01.028082 1541 raft_consensus.cc:2781] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:01.028615 1541 raft_consensus.cc:686] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 1 LEADER]: Becoming Leader. State: Replica: 32b898cf52eb46cc96db13f37367e7fe, State: Running, Role: LEADER
I20260811 03:05:01.029202 1541 consensus_queue.cc:237] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } }
I20260811 03:05:01.034199 995 catalog_manager.cc:5495] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 32b898cf52eb46cc96db13f37367e7fe (127.27.109.1). New cstate: current_term: 1 leader_uuid: "32b898cf52eb46cc96db13f37367e7fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32b898cf52eb46cc96db13f37367e7fe" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 32881 } health_report { overall_health: HEALTHY } } }
I20260811 03:05:01.060822 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:01.058461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.27.109.1:32881)
0811 03:05:01.058735 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:01.058741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:01.058762 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:05:01.059009 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:01.059013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:01.059128 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:01.059240 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:01.060182 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:01.060188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:01.060193 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:01.060212 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:01.060253 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:01.060266 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:01.060395 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:01.060490 (+ 95us) server_negotiation.cc:299] Negotiation successful
0811 03:05:01.060562 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260811 03:05:01.066509 1359 tablet.cc:2369] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:05:01.068866 1207 tablet.cc:2369] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:01.070092 1459 raft_consensus.cc:1261] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 1 FOLLOWER]: Refusing update from remote peer 32b898cf52eb46cc96db13f37367e7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:01.070079 1307 raft_consensus.cc:1261] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 1 FOLLOWER]: Refusing update from remote peer 32b898cf52eb46cc96db13f37367e7fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260811 03:05:01.070372 1511 tablet.cc:2369] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:01.071346 1541 consensus_queue.cc:1035] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "173c1a53b92a4edc95ddf8f5d83e8e62" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:01.071671 1545 consensus_queue.cc:1035] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "19e412d4b89c4cef9b820d349f7ac5ec" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 37741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:01.086813 1549 mvcc.cc:205] Tried to move back new op lower bound from 7317166084371136512 to 7317166084219645952. Current Snapshot: MvccSnapshot[applied={T|T < 7317166084371136512}]
I20260811 03:05:01.093019 1548 mvcc.cc:205] Tried to move back new op lower bound from 7317166084371136512 to 7317166084219645952. Current Snapshot: MvccSnapshot[applied={T|T < 7317166084371136512 or (T in {7317166084371136512})}]
I20260811 03:05:01.103152 993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:01.103502 993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:01.108098 993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 122cfacd7f61481b9e50be759713038e: Sending DeleteTablet for 3 replicas of tablet e62a7bd1296d4f1385fa1ef28a7d2f52
I20260811 03:05:01.109740 1435 tablet_service.cc:1514] Processing DeleteTablet for tablet e62a7bd1296d4f1385fa1ef28a7d2f52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:01 UTC) from {username='slave'} at 127.0.0.1:35618
I20260811 03:05:01.109750 1133 tablet_service.cc:1514] Processing DeleteTablet for tablet e62a7bd1296d4f1385fa1ef28a7d2f52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:01 UTC) from {username='slave'} at 127.0.0.1:51556
I20260811 03:05:01.110052 1285 tablet_service.cc:1514] Processing DeleteTablet for tablet e62a7bd1296d4f1385fa1ef28a7d2f52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:01 UTC) from {username='slave'} at 127.0.0.1:56198
I20260811 03:05:01.112301 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 1063
I20260811 03:05:01.122244 1560 tablet_replica.cc:330] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: stopping tablet replica
I20260811 03:05:01.123045 1561 tablet_replica.cc:330] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: stopping tablet replica
I20260811 03:05:01.123627 1561 raft_consensus.cc:2229] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:05:01.124426 1561 raft_consensus.cc:2258] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:01.125458 1560 raft_consensus.cc:2229] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:01.126068 1560 raft_consensus.cc:2258] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:01.128845 1560 ts_tablet_manager.cc:1905] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:01.131022 1561 ts_tablet_manager.cc:1905] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:01.136829 1560 ts_tablet_manager.cc:1918] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:01.137166 1560 log.cc:1196] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/e62a7bd1296d4f1385fa1ef28a7d2f52
I20260811 03:05:01.137880 1560 ts_tablet_manager.cc:1939] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 19e412d4b89c4cef9b820d349f7ac5ec: Deleting consensus metadata
I20260811 03:05:01.140183 979 catalog_manager.cc:4841] TS 19e412d4b89c4cef9b820d349f7ac5ec (127.27.109.3:37741): tablet e62a7bd1296d4f1385fa1ef28a7d2f52 (table test-table [id=e7916a18b41b4897941d090068b91cf2]) successfully deleted
I20260811 03:05:01.141248 1562 tablet_replica.cc:330] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: stopping tablet replica
I20260811 03:05:01.141305 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:01.115987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.27.109.1:32881)
0811 03:05:01.116781 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:01.116792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:01.138602 (+ 21810us) server_negotiation.cc:407] Connection header received
0811 03:05:01.138893 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:01.138899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:01.138993 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:01.139109 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:01.140394 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:01.140402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:01.140407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:01.140429 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:01.140469 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:01.140483 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:01.140602 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:01.140872 (+ 270us) server_negotiation.cc:299] Negotiation successful
0811 03:05:01.140941 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593}
I20260811 03:05:01.141489 1561 ts_tablet_manager.cc:1918] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:01.141682 1561 log.cc:1196] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/e62a7bd1296d4f1385fa1ef28a7d2f52
I20260811 03:05:01.141840 1562 raft_consensus.cc:2229] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:01.142535 1561 ts_tablet_manager.cc:1939] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 32b898cf52eb46cc96db13f37367e7fe: Deleting consensus metadata
I20260811 03:05:01.142532 1562 raft_consensus.cc:2258] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:01.143123 1201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:01.145473 1562 ts_tablet_manager.cc:1905] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:01.153944 1562 ts_tablet_manager.cc:1918] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:01.154206 1562 log.cc:1196] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/e62a7bd1296d4f1385fa1ef28a7d2f52
I20260811 03:05:01.154876 1562 ts_tablet_manager.cc:1939] T e62a7bd1296d4f1385fa1ef28a7d2f52 P 173c1a53b92a4edc95ddf8f5d83e8e62: Deleting consensus metadata
I20260811 03:05:01.157408 978 catalog_manager.cc:4841] TS 173c1a53b92a4edc95ddf8f5d83e8e62 (127.27.109.2:35565): tablet e62a7bd1296d4f1385fa1ef28a7d2f52 (table test-table [id=e7916a18b41b4897941d090068b91cf2]) successfully deleted
I20260811 03:05:01.249456 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 1063
I20260811 03:05:01.249594 978 catalog_manager.cc:4841] TS 32b898cf52eb46cc96db13f37367e7fe (127.27.109.1:32881): tablet e62a7bd1296d4f1385fa1ef28a7d2f52 (table test-table [id=e7916a18b41b4897941d090068b91cf2]) successfully deleted
I20260811 03:05:01.273660 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 1210
I20260811 03:05:01.277146 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:01.274630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.27.109.2:35565)
0811 03:05:01.274890 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:01.274898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:01.274923 (+ 25us) server_negotiation.cc:407] Connection header received
0811 03:05:01.275078 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:01.275084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:01.275197 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:01.275285 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:01.276370 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:01.276375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:01.276378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:01.276396 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:01.276432 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:01.276483 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:01.276604 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:01.276759 (+ 155us) server_negotiation.cc:299] Negotiation successful
0811 03:05:01.276813 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260811 03:05:01.277856 1353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:01.369364 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 1210
I20260811 03:05:01.393762 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 1362
I20260811 03:05:01.400708 1523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:01.394641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.27.109.3:37741)
0811 03:05:01.394877 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:01.394882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:01.394956 (+ 74us) server_negotiation.cc:407] Connection header received
0811 03:05:01.395190 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:01.395197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:01.395293 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:01.395390 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:01.399922 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:01.399929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:01.399932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:01.399957 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:01.400001 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:01.400016 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:01.400163 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:01.400276 (+ 113us) server_negotiation.cc:299] Negotiation successful
0811 03:05:01.400337 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260811 03:05:01.401476 1504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:01.494215 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 1362
I20260811 03:05:01.516443 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 958
I20260811 03:05:01.517843 1028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:01.630720 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 958
2026-08-11T03:05:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2977 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260811 03:05:01.679701 28084 mini_oidc.cc:152] Starting JWKS server
I20260811 03:05:01.683091 28084 webserver.cc:460] Webserver started at https://127.0.0.1:33971/ using document root <none> and password file <none>
I20260811 03:05:01.683506 28084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 03:05:01.684423 28084 webserver.cc:460] Webserver started at http://127.0.0.1:37931/ using document root <none> and password file <none>
2026-08-11T03:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:01Z Disabled control of system clock
I20260811 03:05:01.726171 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:43979
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:43979
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33971/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/testchainca.cert with env {}
W20260811 03:05:02.098770 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:02.099184 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:02.099285 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:02.107237 1576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:02.107373 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:02.107412 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:02.107479 1576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:02.107524 1576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 03:05:02.107589 1576 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 03:05:02.107630 1576 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33971/jwks.json
I20260811 03:05:02.118289 1576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:43979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33971/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:02.119769 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:02.121141 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:02.134227 1582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:02.134198 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:02.134258 1583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:02.135278 1576 server_base.cc:1034] running on GCE node
I20260811 03:05:02.136319 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:02.137929 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:02.139288 1576 hybrid_clock.cc:648] HybridClock initialized: now 1786417502139190 us; error 94 us; skew 500 ppm
I20260811 03:05:02.139719 1576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:02.142441 1576 webserver.cc:460] Webserver started at http://127.27.109.62:36125/ using document root <none> and password file <none>
I20260811 03:05:02.143309 1576 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:02.143463 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:02.143822 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:02.148974 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4"
format_stamp: "Formatted at 2026-08-11 03:05:02 on dist-test-slave-8tdl"
server_key: "d1ea2b82899dd684901139d556af677e"
server_key_iv: "7553cffcc5ac2194ecac067f5cd7c759"
server_key_version: "encryptionkey@0"
I20260811 03:05:02.150097 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4"
format_stamp: "Formatted at 2026-08-11 03:05:02 on dist-test-slave-8tdl"
server_key: "d1ea2b82899dd684901139d556af677e"
server_key_iv: "7553cffcc5ac2194ecac067f5cd7c759"
server_key_version: "encryptionkey@0"
I20260811 03:05:02.156003 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:05:02.159956 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:02.161538 1576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:05:02.161756 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4"
format_stamp: "Formatted at 2026-08-11 03:05:02 on dist-test-slave-8tdl"
server_key: "d1ea2b82899dd684901139d556af677e"
server_key_iv: "7553cffcc5ac2194ecac067f5cd7c759"
server_key_version: "encryptionkey@0"
I20260811 03:05:02.162088 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:02.177456 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:02.178442 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:02.178851 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:02.198987 1576 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:43979
I20260811 03:05:02.199008 1649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:43979 every 8 connection(s)
I20260811 03:05:02.200559 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:02.206177 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:02.206432 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 1576
I20260811 03:05:02.206646 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:02.207609 28084 external_mini_cluster.cc:1428] Setting key fbc001a8a3b7fcaeba3b13ff7c854d54
I20260811 03:05:02.224257 1650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4: Bootstrap starting.
I20260811 03:05:02.228802 1650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:02.230440 1650 log.cc:824] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:02.232786 1654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:02.212025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.27.109.62:43979)
0811 03:05:02.213793 (+ 1768us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:02.213814 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:02.215420 (+ 1606us) server_negotiation.cc:407] Connection header received
0811 03:05:02.218474 (+ 3054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:02.218519 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:02.219347 (+ 828us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:02.219895 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:02.221375 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:02.223514 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:02.225740 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:02.226975 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:02.227181 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:02.228707 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:02.228721 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:02.228745 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:02.229647 (+ 902us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:02.229795 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:02.229829 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:02.229952 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:02.230720 (+ 768us) server_negotiation.cc:299] Negotiation successful
0811 03:05:02.231119 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":614,"threads_started":1}
I20260811 03:05:02.234170 1650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4: No bootstrap required, opened a new log
I20260811 03:05:02.237860 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:05:02.243875 1650 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43979 } }
I20260811 03:05:02.244510 1650 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:02.244654 1650 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c9cdaa0a404bdfa018d228aed0bfb4, State: Initialized, Role: FOLLOWER
I20260811 03:05:02.245512 1650 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43979 } }
I20260811 03:05:02.245887 1650 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:02.246083 1650 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:02.246313 1650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:02.250583 1650 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43979 } }
I20260811 03:05:02.251492 1650 leader_election.cc:302] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c9cdaa0a404bdfa018d228aed0bfb4; no voters:
I20260811 03:05:02.252350 1650 leader_election.cc:288] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:02.252475 1657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:02.253300 1657 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [term 1 LEADER]: Becoming Leader. State: Replica: d5c9cdaa0a404bdfa018d228aed0bfb4, State: Running, Role: LEADER
I20260811 03:05:02.253965 1657 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43979 } }
I20260811 03:05:02.255158 1650 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:02.257427 1660 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c9cdaa0a404bdfa018d228aed0bfb4. Latest consensus state: current_term: 1 leader_uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43979 } } }
I20260811 03:05:02.258080 1660 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:02.257705 1658 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c9cdaa0a404bdfa018d228aed0bfb4" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43979 } } }
I20260811 03:05:02.258242 1658 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:02.259217 1662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:02.266531 1662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:02.282701 1662 catalog_manager.cc:1347] Generated new cluster ID: e3eb60ae097044109b721ff08aa66438
I20260811 03:05:02.282861 1662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:02.322053 1662 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:02.323643 1662 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:02.331354 1662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c9cdaa0a404bdfa018d228aed0bfb4: Generated new TSK 0
I20260811 03:05:02.332504 1662 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 03:05:02.375795 1687 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260811 03:05:02.376088 1687 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 03:05:02.378664 1654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:02.353851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.27.109.62:43979)
0811 03:05:02.354131 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:02.354138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:02.354523 (+ 385us) server_negotiation.cc:407] Connection header received
0811 03:05:02.355023 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:02.355028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:02.355103 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:02.355256 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0811 03:05:02.355875 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:02.356648 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:02.358118 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:02.358598 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:02.358816 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:02.359385 (+ 569us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 03:05:02.377795 (+ 18410us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 03:05:02.378286 (+ 491us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":91}
W20260811 03:05:02.378847 1654 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260811 03:05:02.380504 1686 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:02.353705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43979 (local address 127.0.0.1:49250)
0811 03:05:02.354408 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:02.354426 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:02.354640 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:02.355289 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:02.355294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:02.355307 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0811 03:05:02.355701 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:02.355711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:02.356800 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:02.356806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:02.357947 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:02.357958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:02.358851 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:02.358858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:02.359013 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:02.359084 (+ 71us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 03:05:02.378184 (+ 19100us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 03:05:02.378577 (+ 393us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.62:43979: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":448,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.109.62:43979: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33971/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260811 03:05:02.382613 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 1576
I20260811 03:05:02.383976 1644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:02.486269 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 1576
2026-08-11T03:05:02Z chronyd exiting
I20260811 03:05:02.541404 28084 test_util.cc:175] -----------------------------------------------
I20260811 03:05:02.541575 28084 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786417454123933-28084-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (867 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260811 03:05:02.545670 28084 mini_oidc.cc:152] Starting JWKS server
I20260811 03:05:02.547879 28084 webserver.cc:460] Webserver started at https://127.0.0.1:36097/ using document root <none> and password file <none>
I20260811 03:05:02.548175 28084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 03:05:02.548905 28084 webserver.cc:460] Webserver started at http://127.0.0.1:45891/ using document root <none> and password file <none>
2026-08-11T03:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:02Z Disabled control of system clock
I20260811 03:05:02.589551 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35005
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:35005
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/testchainca.cert with env {}
W20260811 03:05:02.962442 1696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:02.962873 1696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:02.962983 1696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:02.970981 1696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:02.971115 1696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:02.971155 1696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:02.971206 1696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:02.971246 1696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 03:05:02.971308 1696 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 03:05:02.971343 1696 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36097/jwks.json
I20260811 03:05:02.982599 1696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:35005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:35005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:02.984053 1696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:02.985517 1696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:03.000456 1703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:03.000290 1704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:03.000254 1707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:03.002766 1696 server_base.cc:1034] running on GCE node
I20260811 03:05:03.003983 1696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:03.006174 1696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:03.007563 1696 hybrid_clock.cc:648] HybridClock initialized: now 1786417503007528 us; error 119 us; skew 500 ppm
I20260811 03:05:03.008114 1696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:03.011412 1696 webserver.cc:460] Webserver started at http://127.27.109.62:38115/ using document root <none> and password file <none>
I20260811 03:05:03.012336 1696 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:03.012629 1696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:03.013103 1696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:03.017491 1696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "6a32ce330fd549ebb9a6f677fa31bc16"
format_stamp: "Formatted at 2026-08-11 03:05:03 on dist-test-slave-8tdl"
server_key: "33b60c22d652159127cb070289db3660"
server_key_iv: "7b88f552bf2123efad3afe7eead95437"
server_key_version: "encryptionkey@0"
I20260811 03:05:03.018491 1696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "6a32ce330fd549ebb9a6f677fa31bc16"
format_stamp: "Formatted at 2026-08-11 03:05:03 on dist-test-slave-8tdl"
server_key: "33b60c22d652159127cb070289db3660"
server_key_iv: "7b88f552bf2123efad3afe7eead95437"
server_key_version: "encryptionkey@0"
I20260811 03:05:03.025347 1696 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:05:03.029817 1713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:03.031629 1696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:03.031879 1696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "6a32ce330fd549ebb9a6f677fa31bc16"
format_stamp: "Formatted at 2026-08-11 03:05:03 on dist-test-slave-8tdl"
server_key: "33b60c22d652159127cb070289db3660"
server_key_iv: "7b88f552bf2123efad3afe7eead95437"
server_key_version: "encryptionkey@0"
I20260811 03:05:03.032217 1696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:03.049851 1696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:03.050959 1696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:03.051349 1696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:03.073087 1696 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:35005
I20260811 03:05:03.073107 1771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:35005 every 8 connection(s)
I20260811 03:05:03.074664 1696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:03.077905 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 1696
I20260811 03:05:03.078179 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:03.078958 28084 external_mini_cluster.cc:1428] Setting key 199c2608fc783fbb0de12d28a3f11c4a
I20260811 03:05:03.079221 1772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:03.095845 1772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16: Bootstrap starting.
I20260811 03:05:03.095985 1775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:03.081306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.27.109.62:35005)
0811 03:05:03.082557 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:03.082573 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:03.082632 (+ 59us) server_negotiation.cc:407] Connection header received
0811 03:05:03.084756 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:03.084790 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:03.085512 (+ 722us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:03.086029 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:03.086653 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:03.088756 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:03.090225 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:03.091396 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:03.091599 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:03.092863 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:03.092874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:03.092895 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:03.093698 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:03.093802 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:03.093834 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:03.093938 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:03.094214 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 03:05:03.094576 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":393,"threads_started":1}
I20260811 03:05:03.099570 1772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:03.100344 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:05:03.101035 1772 log.cc:824] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:03.104607 1772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16: No bootstrap required, opened a new log
I20260811 03:05:03.112715 1772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a32ce330fd549ebb9a6f677fa31bc16" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35005 } }
I20260811 03:05:03.113157 1772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:03.113273 1772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a32ce330fd549ebb9a6f677fa31bc16, State: Initialized, Role: FOLLOWER
I20260811 03:05:03.113994 1772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a32ce330fd549ebb9a6f677fa31bc16" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35005 } }
I20260811 03:05:03.114287 1772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:03.114472 1772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:03.114662 1772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:03.117010 1772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a32ce330fd549ebb9a6f677fa31bc16" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35005 } }
I20260811 03:05:03.117601 1772 leader_election.cc:302] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a32ce330fd549ebb9a6f677fa31bc16; no voters:
I20260811 03:05:03.118259 1772 leader_election.cc:288] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:03.118455 1778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:03.119333 1778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [term 1 LEADER]: Becoming Leader. State: Replica: 6a32ce330fd549ebb9a6f677fa31bc16, State: Running, Role: LEADER
I20260811 03:05:03.120042 1778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a32ce330fd549ebb9a6f677fa31bc16" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35005 } }
I20260811 03:05:03.120766 1772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:03.123733 1780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a32ce330fd549ebb9a6f677fa31bc16. Latest consensus state: current_term: 1 leader_uuid: "6a32ce330fd549ebb9a6f677fa31bc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a32ce330fd549ebb9a6f677fa31bc16" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35005 } } }
I20260811 03:05:03.124292 1780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:03.124769 1779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a32ce330fd549ebb9a6f677fa31bc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a32ce330fd549ebb9a6f677fa31bc16" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 35005 } } }
I20260811 03:05:03.125231 1779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:03.127542 1787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:03.133353 1787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:03.143826 1787 catalog_manager.cc:1347] Generated new cluster ID: b005bff5b7224e04b636a55972f7294e
I20260811 03:05:03.143975 1787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:03.165472 1787 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:03.167194 1787 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:03.174185 1787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a32ce330fd549ebb9a6f677fa31bc16: Generated new TSK 0
I20260811 03:05:03.175379 1787 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 03:05:03.216332 1806 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260811 03:05:03.216496 1806 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 03:05:03.217922 1775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:03.188586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.27.109.62:35005)
0811 03:05:03.188790 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:03.188796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:03.189424 (+ 628us) server_negotiation.cc:407] Connection header received
0811 03:05:03.189654 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:03.189662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:03.189778 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:03.189980 (+ 202us) server_negotiation.cc:226] Negotiated authn=JWT
0811 03:05:03.190499 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:03.191727 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:03.193457 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:03.194202 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:03.194350 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:03.195025 (+ 675us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 03:05:03.217009 (+ 21984us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 03:05:03.217555 (+ 546us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260811 03:05:03.218078 1775 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43828: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260811 03:05:03.218111 1805 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:03.188466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:35005 (local address 127.0.0.1:43828)
0811 03:05:03.189284 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:03.189304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:03.189453 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:03.190004 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:03.190007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:03.190019 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0811 03:05:03.190328 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:03.190338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:03.191930 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:03.191935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:03.193277 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:03.193294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:03.194415 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:03.194421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:03.194590 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:03.194632 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 03:05:03.217353 (+ 22721us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 03:05:03.217668 (+ 315us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.62:35005: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":465,"threads_started":1}
I20260811 03:05:03.219583 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 1696
I20260811 03:05:03.221024 1766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:03.318135 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 1696
2026-08-11T03:05:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (832 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260811 03:05:03.378319 28084 mini_oidc.cc:152] Starting JWKS server
I20260811 03:05:03.380503 28084 webserver.cc:460] Webserver started at https://127.0.0.1:36039/ using document root <none> and password file <none>
I20260811 03:05:03.380823 28084 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 03:05:03.381716 28084 webserver.cc:460] Webserver started at http://127.0.0.1:33407/ using document root <none> and password file <none>
2026-08-11T03:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:03Z Disabled control of system clock
I20260811 03:05:03.423012 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37827
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:37827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36039/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 03:05:03.790530 1813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:03.790980 1813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:03.791162 1813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:03.799110 1813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:03.799314 1813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:03.799422 1813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:03.799515 1813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:03.799642 1813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 03:05:03.799741 1813 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 03:05:03.799830 1813 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36039/jwks.json
I20260811 03:05:03.809826 1813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:37827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36039/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:03.811406 1813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:03.813357 1813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:03.826596 1820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:03.826565 1821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:03.826661 1824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:03.827462 1813 server_base.cc:1034] running on GCE node
I20260811 03:05:03.828505 1813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:03.830116 1813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:03.831328 1813 hybrid_clock.cc:648] HybridClock initialized: now 1786417503831292 us; error 31 us; skew 500 ppm
I20260811 03:05:03.831719 1813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:03.834394 1813 webserver.cc:460] Webserver started at http://127.27.109.62:42769/ using document root <none> and password file <none>
I20260811 03:05:03.835271 1813 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:03.835368 1813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:03.835841 1813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:03.839437 1813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "395d2e51805b4839a7127fe48fec725c"
format_stamp: "Formatted at 2026-08-11 03:05:03 on dist-test-slave-8tdl"
server_key: "63e2efe5a26694cbc38e7525ddad9d9a"
server_key_iv: "cb6b7eb42d5230da0b2481762741e871"
server_key_version: "encryptionkey@0"
I20260811 03:05:03.840332 1813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "395d2e51805b4839a7127fe48fec725c"
format_stamp: "Formatted at 2026-08-11 03:05:03 on dist-test-slave-8tdl"
server_key: "63e2efe5a26694cbc38e7525ddad9d9a"
server_key_iv: "cb6b7eb42d5230da0b2481762741e871"
server_key_version: "encryptionkey@0"
I20260811 03:05:03.846314 1813 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:05:03.850382 1832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:03.852172 1813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:03.852401 1813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "395d2e51805b4839a7127fe48fec725c"
format_stamp: "Formatted at 2026-08-11 03:05:03 on dist-test-slave-8tdl"
server_key: "63e2efe5a26694cbc38e7525ddad9d9a"
server_key_iv: "cb6b7eb42d5230da0b2481762741e871"
server_key_version: "encryptionkey@0"
I20260811 03:05:03.852658 1813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:03.887877 1813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:03.888820 1813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:03.889161 1813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:03.908931 1813 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:37827
I20260811 03:05:03.908949 1887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:37827 every 8 connection(s)
I20260811 03:05:03.910480 1813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:03.912721 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 1813
I20260811 03:05:03.912926 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:03.913617 28084 external_mini_cluster.cc:1428] Setting key 49c8c5cf884cbee1e9a45f0ff787b7b0
I20260811 03:05:03.915302 1888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:03.930665 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:03.915808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.27.109.62:37827)
0811 03:05:03.916967 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:03.916983 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:03.917039 (+ 56us) server_negotiation.cc:407] Connection header received
0811 03:05:03.919003 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:03.919036 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:03.919792 (+ 756us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:03.920322 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:03.920962 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:03.923184 (+ 2222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:03.924851 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:03.925946 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:03.926151 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:03.927491 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:03.927504 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:03.927532 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:03.928375 (+ 843us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:03.928486 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:03.928515 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:03.928699 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:03.928853 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 03:05:03.929248 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":358,"threads_started":1}
I20260811 03:05:03.931543 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c: Bootstrap starting.
I20260811 03:05:03.935125 1888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:03.935225 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:05:03.936208 1888 log.cc:824] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:03.939704 1888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c: No bootstrap required, opened a new log
I20260811 03:05:03.948323 1888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d2e51805b4839a7127fe48fec725c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37827 } }
I20260811 03:05:03.948770 1888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:03.948911 1888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395d2e51805b4839a7127fe48fec725c, State: Initialized, Role: FOLLOWER
I20260811 03:05:03.949533 1888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d2e51805b4839a7127fe48fec725c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37827 } }
I20260811 03:05:03.949787 1888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:03.949955 1888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:03.950150 1888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:03.952661 1888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d2e51805b4839a7127fe48fec725c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37827 } }
I20260811 03:05:03.953307 1888 leader_election.cc:302] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 395d2e51805b4839a7127fe48fec725c; no voters:
I20260811 03:05:03.954064 1888 leader_election.cc:288] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:03.954188 1893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:03.955195 1893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [term 1 LEADER]: Becoming Leader. State: Replica: 395d2e51805b4839a7127fe48fec725c, State: Running, Role: LEADER
I20260811 03:05:03.955806 1893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d2e51805b4839a7127fe48fec725c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37827 } }
I20260811 03:05:03.956549 1888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:03.959045 1895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "395d2e51805b4839a7127fe48fec725c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d2e51805b4839a7127fe48fec725c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37827 } } }
I20260811 03:05:03.959303 1896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 395d2e51805b4839a7127fe48fec725c. Latest consensus state: current_term: 1 leader_uuid: "395d2e51805b4839a7127fe48fec725c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395d2e51805b4839a7127fe48fec725c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37827 } } }
I20260811 03:05:03.959580 1895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:03.959637 1896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:03.961300 1904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:03.966140 1904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:03.975286 1904 catalog_manager.cc:1347] Generated new cluster ID: 9220d272465a445f934c7b983bb080d9
I20260811 03:05:03.975435 1904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:03.999614 1904 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:04.001672 1904 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:04.010833 1904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 395d2e51805b4839a7127fe48fec725c: Generated new TSK 0
I20260811 03:05:04.012183 1904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:05:04.041874 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:04.032788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.27.109.62:37827)
0811 03:05:04.032987 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:04.032994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:04.033482 (+ 488us) server_negotiation.cc:407] Connection header received
0811 03:05:04.033758 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:04.033763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:04.033853 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:04.034033 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0811 03:05:04.034512 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.035726 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.037240 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.037950 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.038109 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:04.038713 (+ 604us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 03:05:04.041187 (+ 2474us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 03:05:04.041541 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49282: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":77}
W20260811 03:05:04.042011 1891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49282: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260811 03:05:04.043640 1920 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:04.032646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37827 (local address 127.0.0.1:49282)
0811 03:05:04.033362 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:04.033383 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:04.033515 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:04.034064 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:04.034068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:04.034078 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0811 03:05:04.034360 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:04.034367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.035928 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.035933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:04.037082 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:04.037096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.038174 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.038179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:04.038318 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:04.038352 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 03:05:04.041442 (+ 3090us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 03:05:04.043159 (+ 1717us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.62:37827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36039/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":1381,"thread_start_us":414,"threads_started":1}
I20260811 03:05:04.045521 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 1813
I20260811 03:05:04.047051 1883 generic_service.cc:196] Checking for leaks (request via RPC)
E20260811 03:05:04.054451 1921 webserver.cc:533] Webserver: sslize failed:
E20260811 03:05:04.054606 1921 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 03:05:04.142796 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 1813
2026-08-11T03:05:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (825 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-11T03:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:04Z Disabled control of system clock
I20260811 03:05:04.246169 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42425
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:40883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:42425
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/testchainca.cert with env {}
W20260811 03:05:04.626293 1926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:04.626657 1926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:04.626860 1926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:04.634558 1926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:04.634754 1926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:04.634862 1926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:04.634935 1926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:04.634984 1926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 03:05:04.635051 1926 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 03:05:04.635087 1926 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260811 03:05:04.645951 1926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:42425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:04.647428 1926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:04.648820 1926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:04.662358 1933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:04.662391 1931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:04.662526 1936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:04.663314 1926 server_base.cc:1034] running on GCE node
I20260811 03:05:04.664255 1926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:04.665875 1926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:04.667133 1926 hybrid_clock.cc:648] HybridClock initialized: now 1786417504667073 us; error 63 us; skew 500 ppm
I20260811 03:05:04.667546 1926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:04.670166 1926 webserver.cc:460] Webserver started at http://127.27.109.62:35573/ using document root <none> and password file <none>
I20260811 03:05:04.671069 1926 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:04.671223 1926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:04.671568 1926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:04.674924 1926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "2b87fe47c9a3483daa4e2c07ba3af73a"
format_stamp: "Formatted at 2026-08-11 03:05:04 on dist-test-slave-8tdl"
server_key: "d9e1986b708a633e7f27528da337e14d"
server_key_iv: "f506a083480291b379cc4c2e87285a39"
server_key_version: "encryptionkey@0"
I20260811 03:05:04.675886 1926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "2b87fe47c9a3483daa4e2c07ba3af73a"
format_stamp: "Formatted at 2026-08-11 03:05:04 on dist-test-slave-8tdl"
server_key: "d9e1986b708a633e7f27528da337e14d"
server_key_iv: "f506a083480291b379cc4c2e87285a39"
server_key_version: "encryptionkey@0"
I20260811 03:05:04.683712 1926 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260811 03:05:04.688023 1944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:04.689678 1926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:05:04.689929 1926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "2b87fe47c9a3483daa4e2c07ba3af73a"
format_stamp: "Formatted at 2026-08-11 03:05:04 on dist-test-slave-8tdl"
server_key: "d9e1986b708a633e7f27528da337e14d"
server_key_iv: "f506a083480291b379cc4c2e87285a39"
server_key_version: "encryptionkey@0"
I20260811 03:05:04.690209 1926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:04.709199 1926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:04.710258 1926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:04.710597 1926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:04.732143 1926 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:42425
I20260811 03:05:04.732265 2002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:42425 every 8 connection(s)
I20260811 03:05:04.733863 1926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:04.736388 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 1926
I20260811 03:05:04.736666 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:04.737557 28084 external_mini_cluster.cc:1428] Setting key f3cbb2415aa04914550d78a7891dcb67
I20260811 03:05:04.740244 2003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:04.756454 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:04.740481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.27.109.62:42425)
0811 03:05:04.741723 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:04.741742 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:04.741806 (+ 64us) server_negotiation.cc:407] Connection header received
0811 03:05:04.744053 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:04.744091 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:04.744885 (+ 794us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:04.745487 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:04.746138 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.748426 (+ 2288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.750073 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.751273 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.751557 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:04.753009 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:04.753022 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:04.753045 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:04.753923 (+ 878us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:04.754034 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:04.754071 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:04.754200 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:04.754527 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 03:05:04.754979 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":399,"threads_started":1}
I20260811 03:05:04.757203 2003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a: Bootstrap starting.
I20260811 03:05:04.760856 2003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:04.761052 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:05:04.762303 2003 log.cc:824] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:04.765646 2003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a: No bootstrap required, opened a new log
I20260811 03:05:04.773679 2003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b87fe47c9a3483daa4e2c07ba3af73a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42425 } }
I20260811 03:05:04.774178 2003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:04.774341 2003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b87fe47c9a3483daa4e2c07ba3af73a, State: Initialized, Role: FOLLOWER
I20260811 03:05:04.775070 2003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b87fe47c9a3483daa4e2c07ba3af73a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42425 } }
I20260811 03:05:04.775357 2003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:04.775527 2003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:04.775727 2003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:04.778069 2003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b87fe47c9a3483daa4e2c07ba3af73a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42425 } }
I20260811 03:05:04.778749 2003 leader_election.cc:302] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b87fe47c9a3483daa4e2c07ba3af73a; no voters:
I20260811 03:05:04.779531 2003 leader_election.cc:288] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:04.779779 2008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:04.780942 2008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [term 1 LEADER]: Becoming Leader. State: Replica: 2b87fe47c9a3483daa4e2c07ba3af73a, State: Running, Role: LEADER
I20260811 03:05:04.781601 2008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b87fe47c9a3483daa4e2c07ba3af73a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42425 } }
I20260811 03:05:04.782042 2003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:04.784636 2011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b87fe47c9a3483daa4e2c07ba3af73a. Latest consensus state: current_term: 1 leader_uuid: "2b87fe47c9a3483daa4e2c07ba3af73a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b87fe47c9a3483daa4e2c07ba3af73a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42425 } } }
I20260811 03:05:04.785307 2011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:04.785023 2010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b87fe47c9a3483daa4e2c07ba3af73a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b87fe47c9a3483daa4e2c07ba3af73a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42425 } } }
I20260811 03:05:04.785466 2010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:04.789004 2019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:04.795284 2019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:04.806496 2019 catalog_manager.cc:1347] Generated new cluster ID: 44a0ce30f2264725a0d393708b59fd91
I20260811 03:05:04.806655 2019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:04.834187 2019 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:04.835950 2019 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:04.847755 2019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b87fe47c9a3483daa4e2c07ba3af73a: Generated new TSK 0
I20260811 03:05:04.848841 2019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:05:05.310452 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:04.868282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.27.109.62:42425)
0811 03:05:04.868602 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:04.868610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:04.868959 (+ 349us) server_negotiation.cc:407] Connection header received
0811 03:05:04.869293 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:04.869298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:04.869379 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:04.869585 (+ 206us) server_negotiation.cc:226] Negotiated authn=JWT
0811 03:05:04.870149 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.871062 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.872694 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.873233 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.873389 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:04.874025 (+ 636us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 03:05:05.309632 (+435607us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 03:05:05.310124 (+ 492us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40870: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":116}
W20260811 03:05:05.310603 2006 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40870: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260811 03:05:05.310783 2035 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:04.868097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42425 (local address 127.0.0.1:40870)
0811 03:05:04.868833 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:04.868851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:04.869053 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:04.869601 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:04.869606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:04.869620 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0811 03:05:04.869953 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:04.869965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.871241 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.871255 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:04.872512 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:04.872526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:04.873459 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:04.873469 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:04.873634 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:04.873676 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 03:05:05.309971 (+436295us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 03:05:05.310308 (+ 337us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.109.62:42425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":420,"threads_started":1}
I20260811 03:05:05.312081 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 1926
I20260811 03:05:05.313329 1998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:05.424220 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 1926
2026-08-11T03:05:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1263 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (458 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (457 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:05:06 dist-test-slave-8tdl krb5kdc[2054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:05:06 dist-test-slave-8tdl krb5kdc[2054](info): set up 2 sockets
Aug 11 03:05:06 dist-test-slave-8tdl krb5kdc[2054](info): commencing operation
krb5kdc: starting...
W20260811 03:05:07.554095 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:05:07 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:05:07.844843 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:40067
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:40067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:08.211701 2068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:08.212033 2068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:08.212198 2068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:08.219866 2068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:08.220005 2068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:08.220098 2068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:08.220198 2068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:08.220278 2068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:08.230890 2068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:40067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:08.232439 2068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:08.233860 2068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:08.246662 2076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:08.246630 2080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:08.246742 2077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:08.247841 2068 server_base.cc:1034] running on GCE node
I20260811 03:05:08.248769 2068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:08.250379 2068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:08.251659 2068 hybrid_clock.cc:648] HybridClock initialized: now 1786417508251578 us; error 65 us; skew 500 ppm
I20260811 03:05:08.252130 2068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:08 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:08.255812 2068 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:05:08.257638 2068 webserver.cc:460] Webserver started at http://127.27.109.62:33037/ using document root <none> and password file <none>
I20260811 03:05:08.258428 2068 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:08.258591 2068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:08.258963 2068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:08.262624 2068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "ebe14fcc64184dbfa25a3c25415f0263"
format_stamp: "Formatted at 2026-08-11 03:05:08 on dist-test-slave-8tdl"
server_key: "39c33b3ee2b6108ad2d3a829822a98fe"
server_key_iv: "52cde3c5933fb17e3627e80d1031955b"
server_key_version: "encryptionkey@0"
I20260811 03:05:08.263679 2068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "ebe14fcc64184dbfa25a3c25415f0263"
format_stamp: "Formatted at 2026-08-11 03:05:08 on dist-test-slave-8tdl"
server_key: "39c33b3ee2b6108ad2d3a829822a98fe"
server_key_iv: "52cde3c5933fb17e3627e80d1031955b"
server_key_version: "encryptionkey@0"
I20260811 03:05:08.269876 2068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:05:08.274309 2088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:08.276298 2068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:05:08.276557 2068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "ebe14fcc64184dbfa25a3c25415f0263"
format_stamp: "Formatted at 2026-08-11 03:05:08 on dist-test-slave-8tdl"
server_key: "39c33b3ee2b6108ad2d3a829822a98fe"
server_key_iv: "52cde3c5933fb17e3627e80d1031955b"
server_key_version: "encryptionkey@0"
I20260811 03:05:08.276822 2068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:08.313144 2068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:08.318723 2068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:08.319096 2068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:08.340309 2068 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:40067
I20260811 03:05:08.340350 2147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:40067 every 8 connection(s)
I20260811 03:05:08.341933 2068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:08.344159 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 2068
I20260811 03:05:08.344432 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:08.345144 28084 external_mini_cluster.cc:1428] Setting key 13e91114c89c3aa0f8f98203a800b2d4
I20260811 03:05:08.346799 2148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 03:05:08 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:05:08.362962 2148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263: Bootstrap starting.
I20260811 03:05:08.367637 2148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:08.369015 2148 log.cc:824] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:08.372514 2148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263: No bootstrap required, opened a new log
I20260811 03:05:08.373095 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:08.347552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.27.109.62:40067)
0811 03:05:08.348472 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:08.348490 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:08.348555 (+ 65us) server_negotiation.cc:407] Connection header received
0811 03:05:08.351038 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:08.351073 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:08.351746 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:08.352270 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:08.353685 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:08.355930 (+ 2245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:08.357401 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:08.358507 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:08.358753 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:08.363775 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:08.363788 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:08.363810 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:08.363870 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:08.368321 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:08.369175 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:08.369184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:08.369193 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:08.369427 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:08.369886 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:08.369891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:08.369893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:08.370579 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:08.370971 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:08.371451 (+ 480us) server_negotiation.cc:299] Negotiation successful
0811 03:05:08.371928 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":364,"threads_started":1}
I20260811 03:05:08.382508 2148 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe14fcc64184dbfa25a3c25415f0263" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40067 } }
I20260811 03:05:08.383137 2148 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:08.383280 2148 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe14fcc64184dbfa25a3c25415f0263, State: Initialized, Role: FOLLOWER
I20260811 03:05:08.384023 2148 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe14fcc64184dbfa25a3c25415f0263" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40067 } }
I20260811 03:05:08.384261 2148 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:08.384364 2148 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:08.384506 2148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:08.386914 2148 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe14fcc64184dbfa25a3c25415f0263" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40067 } }
I20260811 03:05:08.387542 2148 leader_election.cc:302] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe14fcc64184dbfa25a3c25415f0263; no voters:
I20260811 03:05:08.388269 2148 leader_election.cc:288] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:08.388414 2153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:08.390877 2148 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:08.391314 2153 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [term 1 LEADER]: Becoming Leader. State: Replica: ebe14fcc64184dbfa25a3c25415f0263, State: Running, Role: LEADER
I20260811 03:05:08.391891 2153 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe14fcc64184dbfa25a3c25415f0263" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40067 } }
I20260811 03:05:08.395320 2155 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe14fcc64184dbfa25a3c25415f0263. Latest consensus state: current_term: 1 leader_uuid: "ebe14fcc64184dbfa25a3c25415f0263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe14fcc64184dbfa25a3c25415f0263" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40067 } } }
I20260811 03:05:08.395903 2155 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:08.395988 2154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebe14fcc64184dbfa25a3c25415f0263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe14fcc64184dbfa25a3c25415f0263" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40067 } } }
I20260811 03:05:08.396443 2154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:08.398061 2166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:08.402940 2166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:08.413410 2166 catalog_manager.cc:1347] Generated new cluster ID: 3564120666084215b003ae4c775e0df9
I20260811 03:05:08.413580 2166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:08.464771 2166 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:08.467186 2166 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:08.478972 2166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebe14fcc64184dbfa25a3c25415f0263: Generated new TSK 0
I20260811 03:05:08.480248 2166 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:05:08.555043 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40067
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:08.933830 2177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:08.934181 2177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:08.934352 2177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:08.942252 2177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:08.942384 2177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:08.942577 2177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:05:08.953243 2177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:08.954780 2177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:08.956411 2177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:08.971604 2184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:08.971678 2185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:08.971647 2187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:08.972353 2177 server_base.cc:1034] running on GCE node
I20260811 03:05:08.973273 2177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:08.974406 2177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:08.975692 2177 hybrid_clock.cc:648] HybridClock initialized: now 1786417508975618 us; error 62 us; skew 500 ppm
I20260811 03:05:08.976090 2177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:08 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:08.980180 2177 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:05:08.982425 2177 webserver.cc:460] Webserver started at http://127.27.109.1:42971/ using document root <none> and password file <none>
I20260811 03:05:08.983393 2177 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:08.983559 2177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:08.983930 2177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:08.987389 2177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "434a7bea42b74671b97da0df0baec64b"
format_stamp: "Formatted at 2026-08-11 03:05:08 on dist-test-slave-8tdl"
server_key: "106dedada5a41c8199edfd274d67e57a"
server_key_iv: "0210c889cc31968cafcb1209095211cf"
server_key_version: "encryptionkey@0"
I20260811 03:05:08.988368 2177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "434a7bea42b74671b97da0df0baec64b"
format_stamp: "Formatted at 2026-08-11 03:05:08 on dist-test-slave-8tdl"
server_key: "106dedada5a41c8199edfd274d67e57a"
server_key_iv: "0210c889cc31968cafcb1209095211cf"
server_key_version: "encryptionkey@0"
I20260811 03:05:08.995088 2177 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 03:05:08.999521 2197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:09.001257 2177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:05:09.001497 2177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "434a7bea42b74671b97da0df0baec64b"
format_stamp: "Formatted at 2026-08-11 03:05:08 on dist-test-slave-8tdl"
server_key: "106dedada5a41c8199edfd274d67e57a"
server_key_iv: "0210c889cc31968cafcb1209095211cf"
server_key_version: "encryptionkey@0"
I20260811 03:05:09.001765 2177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:09.024315 2177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:09.028915 2177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:09.029197 2177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:09.030252 2177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:09.031831 2177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:09.031913 2177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:09.032008 2177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:09.032059 2177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:09.069742 2177 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:44345
I20260811 03:05:09.069857 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:44345 every 8 connection(s)
I20260811 03:05:09.071214 2177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:05:09.075188 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 2177
I20260811 03:05:09.075423 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:05:09.076144 28084 external_mini_cluster.cc:1428] Setting key 3a47c7878f8e36abb3c7d70d674dcf50
Aug 11 03:05:09 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:05:09.098744 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:09.073963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:46887 (local address 127.27.109.62:40067)
0811 03:05:09.074424 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:09.074428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:09.075678 (+ 1250us) server_negotiation.cc:407] Connection header received
0811 03:05:09.077885 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:09.077889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:09.077965 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:09.078219 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:09.080330 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:09.081605 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:09.083986 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:09.084730 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:09.084896 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:09.090234 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:09.090245 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:09.090254 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:09.090311 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:09.094479 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:09.095631 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:09.095640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:09.095646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:09.095828 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:09.096430 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:09.096438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:09.096444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:09.096898 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:09.097095 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:09.097873 (+ 778us) server_negotiation.cc:299] Negotiation successful
0811 03:05:09.098164 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260811 03:05:09.099776 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:09.074540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:40067 (local address 127.27.109.1:46887)
0811 03:05:09.075490 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:09.075545 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:09.077580 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:09.078598 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:09.078619 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:09.079431 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:09.080103 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:09.080123 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:09.081800 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:09.081804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:09.083788 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:09.083809 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:09.085005 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:09.085013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:09.085392 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:09.086423 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:09.086464 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:09.089863 (+ 3399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:09.094759 (+ 4896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:09.094769 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:09.094786 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:09.095451 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:09.096040 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:09.096044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:09.096046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:09.096280 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:09.097062 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:09.097069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:09.097443 (+ 374us) client_negotiation.cc:769] Sending connection context
0811 03:05:09.097715 (+ 272us) client_negotiation.cc:240] Negotiation successful
0811 03:05:09.098181 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":457,"threads_started":1}
I20260811 03:05:09.101555 2328 heartbeater.cc:347] Connected to a master server at 127.27.109.62:40067
I20260811 03:05:09.101907 2328 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:09.102772 2328 heartbeater.cc:511] Master 127.27.109.62:40067 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260811 03:05:09.105079 2109 ts_manager.cc:194] Registered new tserver with Master: 434a7bea42b74671b97da0df0baec64b (127.27.109.1:44345)
I20260811 03:05:09.106920 2109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:46887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:05:09.140725 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40067
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:09.516817 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:09.517138 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:09.517298 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:09.524978 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:09.525120 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:09.525359 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:09.535853 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:09.537416 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:09.539016 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:09.554451 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:09.554525 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:09.554551 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:09.555559 2334 server_base.cc:1034] running on GCE node
I20260811 03:05:09.556237 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:09.557263 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:09.558504 2334 hybrid_clock.cc:648] HybridClock initialized: now 1786417509558441 us; error 48 us; skew 500 ppm
I20260811 03:05:09.558925 2334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:09 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417509, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:09.562762 2334 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:05:09.564677 2334 webserver.cc:460] Webserver started at http://127.27.109.2:40843/ using document root <none> and password file <none>
I20260811 03:05:09.565551 2334 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:09.565716 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:09.566064 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:09.569638 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "6835d581648f4bebad91f75f6650d874"
format_stamp: "Formatted at 2026-08-11 03:05:09 on dist-test-slave-8tdl"
server_key: "652f18489333550026f84d71b2a506aa"
server_key_iv: "048aba7b631ec0b014b841479bf48b35"
server_key_version: "encryptionkey@0"
I20260811 03:05:09.570585 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "6835d581648f4bebad91f75f6650d874"
format_stamp: "Formatted at 2026-08-11 03:05:09 on dist-test-slave-8tdl"
server_key: "652f18489333550026f84d71b2a506aa"
server_key_iv: "048aba7b631ec0b014b841479bf48b35"
server_key_version: "encryptionkey@0"
I20260811 03:05:09.577402 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260811 03:05:09.582266 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:09.584367 2334 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260811 03:05:09.584820 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "6835d581648f4bebad91f75f6650d874"
format_stamp: "Formatted at 2026-08-11 03:05:09 on dist-test-slave-8tdl"
server_key: "652f18489333550026f84d71b2a506aa"
server_key_iv: "048aba7b631ec0b014b841479bf48b35"
server_key_version: "encryptionkey@0"
I20260811 03:05:09.585172 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:09.602437 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:09.608433 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:09.608770 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:09.609872 2334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:09.611477 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:09.611562 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:09.611655 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:09.611706 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:09.648238 2334 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:41583
I20260811 03:05:09.648275 2482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:41583 every 8 connection(s)
I20260811 03:05:09.649786 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:09.659835 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 2334
I20260811 03:05:09.660090 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:05:09.660569 28084 external_mini_cluster.cc:1428] Setting key 4f053262b9197f2a0cd2675b988f2c80
Aug 11 03:05:09 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417509, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:05:09.673254 2485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:09.652642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:59873 (local address 127.27.109.62:40067)
0811 03:05:09.653240 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:09.653245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:09.654114 (+ 869us) server_negotiation.cc:407] Connection header received
0811 03:05:09.656143 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:09.656149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:09.656258 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:09.656381 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:09.658094 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:09.658893 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:09.660767 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:09.661454 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:09.661620 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:09.665648 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:09.665658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:09.665667 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:09.665717 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:09.669699 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:09.670606 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:09.670615 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:09.670621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:09.670832 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:09.671307 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:09.671315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:09.671321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:09.671759 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:09.671930 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:09.672714 (+ 784us) server_negotiation.cc:299] Negotiation successful
0811 03:05:09.672988 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":381,"threads_started":1}
I20260811 03:05:09.674916 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:09.653126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:40067 (local address 127.27.109.2:59873)
0811 03:05:09.653889 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:09.653935 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:09.655830 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:09.656660 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:09.656669 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:09.657251 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:09.657882 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:09.657901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:09.659076 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:09.659080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:09.660565 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:09.660578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:09.661703 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:09.661709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:09.662050 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:09.662903 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:09.662929 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:09.665337 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:09.669907 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:09.669916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:09.669934 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:09.670455 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:09.670959 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:09.670964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:09.670967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:09.671187 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:09.671911 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:09.671918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:09.672306 (+ 388us) client_negotiation.cc:769] Sending connection context
0811 03:05:09.672570 (+ 264us) client_negotiation.cc:240] Negotiation successful
0811 03:05:09.673171 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":315,"threads_started":1}
I20260811 03:05:09.676728 2483 heartbeater.cc:347] Connected to a master server at 127.27.109.62:40067
I20260811 03:05:09.677122 2483 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:09.678007 2483 heartbeater.cc:511] Master 127.27.109.62:40067 requested a full tablet report, sending...
I20260811 03:05:09.679790 2109 ts_manager.cc:194] Registered new tserver with Master: 6835d581648f4bebad91f75f6650d874 (127.27.109.2:41583)
I20260811 03:05:09.680853 2109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:59873
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:05:09.726476 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40067
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:10.101648 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:10.102149 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:10.102339 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 03:05:10.110333 2328 heartbeater.cc:503] Master 127.27.109.62:40067 was elected leader, sending a full tablet report...
W20260811 03:05:10.110864 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:10.110973 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:10.111156 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:10.122136 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:10.123724 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:10.125165 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:10.140198 2496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:10.140357 2499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:10.140581 2497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:10.141322 2489 server_base.cc:1034] running on GCE node
I20260811 03:05:10.142026 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:10.143160 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:10.144384 2489 hybrid_clock.cc:648] HybridClock initialized: now 1786417510144347 us; error 39 us; skew 500 ppm
I20260811 03:05:10.144771 2489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:10.148479 2489 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:05:10.150380 2489 webserver.cc:460] Webserver started at http://127.27.109.3:45335/ using document root <none> and password file <none>
I20260811 03:05:10.151250 2489 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:10.151368 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:10.151741 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:10.155068 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "82698fa55263478fbd2b27780113ea4f"
format_stamp: "Formatted at 2026-08-11 03:05:10 on dist-test-slave-8tdl"
server_key: "44fabb867ceaa75483640d341bd9b4c1"
server_key_iv: "e5a53a0cd68da8c6c7f7c7f400fa3978"
server_key_version: "encryptionkey@0"
I20260811 03:05:10.155893 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "82698fa55263478fbd2b27780113ea4f"
format_stamp: "Formatted at 2026-08-11 03:05:10 on dist-test-slave-8tdl"
server_key: "44fabb867ceaa75483640d341bd9b4c1"
server_key_iv: "e5a53a0cd68da8c6c7f7c7f400fa3978"
server_key_version: "encryptionkey@0"
I20260811 03:05:10.161967 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:05:10.167137 2506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:10.168979 2489 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:05:10.169209 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "82698fa55263478fbd2b27780113ea4f"
format_stamp: "Formatted at 2026-08-11 03:05:10 on dist-test-slave-8tdl"
server_key: "44fabb867ceaa75483640d341bd9b4c1"
server_key_iv: "e5a53a0cd68da8c6c7f7c7f400fa3978"
server_key_version: "encryptionkey@0"
I20260811 03:05:10.169487 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:10.186303 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:10.192224 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:10.192612 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:10.193758 2489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:10.195497 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:10.195582 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:10.195737 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:10.195796 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:10.232846 2489 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:41999
I20260811 03:05:10.232892 2635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:41999 every 8 connection(s)
I20260811 03:05:10.234297 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:10.237380 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 2489
I20260811 03:05:10.237625 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:05:10.238169 28084 external_mini_cluster.cc:1428] Setting key 6ed091ac56c08d7ea94e271e31f39eeb
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:05:10.259686 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:10.237438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:43273 (local address 127.27.109.62:40067)
0811 03:05:10.238081 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.238088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.239352 (+ 1264us) server_negotiation.cc:407] Connection header received
0811 03:05:10.242087 (+ 2735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.242091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.242181 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.242291 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.244338 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.245286 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.247398 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.247887 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.248016 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:10.252275 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:10.252281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:10.252284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:10.252326 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:10.255970 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.256921 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.256926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.256928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.257080 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.257611 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.257615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.257617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.257952 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:10.258097 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:10.259050 (+ 953us) server_negotiation.cc:299] Negotiation successful
0811 03:05:10.259323 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":306,"threads_started":1}
I20260811 03:05:10.260804 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:10.238076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:40067 (local address 127.27.109.3:43273)
0811 03:05:10.239146 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:10.239210 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:10.241658 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:10.242559 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:10.242569 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:10.243393 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:10.244105 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:10.244129 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.245554 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.245559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:10.247203 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:10.247221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.248251 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.248257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:10.248519 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:10.249285 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:10.249312 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:10.251883 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:10.256163 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:10.256173 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:10.256191 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:10.256753 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:10.257215 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:10.257219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:10.257222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:10.257478 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:10.258119 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:10.258126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:10.258528 (+ 402us) client_negotiation.cc:769] Sending connection context
0811 03:05:10.258883 (+ 355us) client_negotiation.cc:240] Negotiation successful
0811 03:05:10.259375 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":374,"threads_started":1}
I20260811 03:05:10.262495 2636 heartbeater.cc:347] Connected to a master server at 127.27.109.62:40067
I20260811 03:05:10.262926 2636 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:10.263792 2636 heartbeater.cc:511] Master 127.27.109.62:40067 requested a full tablet report, sending...
I20260811 03:05:10.265579 2109 ts_manager.cc:194] Registered new tserver with Master: 82698fa55263478fbd2b27780113ea4f (127.27.109.3:41999)
I20260811 03:05:10.266657 2109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:43273
I20260811 03:05:10.268183 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/test-admin.keytab.
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:10.301921 28084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260811 03:05:10.308279 2644 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:10.305220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.27.109.1:44345)
0811 03:05:10.305907 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.305914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.305942 (+ 28us) server_negotiation.cc:407] Connection header received
0811 03:05:10.306009 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.306018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.306237 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.306376 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.307773 (+ 1397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36464: BlockingRecv error: recv got EOF from 127.0.0.1:36464 (error 108)
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":350,"threads_started":1}
W20260811 03:05:10.308646 28084 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5cc' of type FILE
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:10.310665 28084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:10.312214 28084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:05:10.330758 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:10.314009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.27.109.1:44345)
0811 03:05:10.314294 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.314300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.314556 (+ 256us) server_negotiation.cc:407] Connection header received
0811 03:05:10.314805 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.314810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.314943 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.315042 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.316267 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.317434 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.318813 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.319424 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.319534 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:10.323739 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:10.323756 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:10.323773 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:10.323816 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:10.327288 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.328151 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.328160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.328163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.328309 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.328833 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.328838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.328840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.329359 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:10.329603 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:10.330009 (+ 406us) server_negotiation.cc:299] Negotiation successful
0811 03:05:10.330367 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
W20260811 03:05:10.337689 2644 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:10.335588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.27.109.1:44345)
0811 03:05:10.335842 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.335849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.336107 (+ 258us) server_negotiation.cc:407] Connection header received
0811 03:05:10.336274 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.336278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.336355 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.336451 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.337404 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36480: BlockingRecv error: recv got EOF from 127.0.0.1:36480 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260811 03:05:10.338230 28084 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5cc)
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:10.340128 28084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:10.341660 28084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:05:10.358386 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:10.343494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.27.109.1:44345)
0811 03:05:10.343720 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.343726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.343863 (+ 137us) server_negotiation.cc:407] Connection header received
0811 03:05:10.344145 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.344150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.344216 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.344323 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.345355 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.346109 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.347707 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.348174 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.348320 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:10.352691 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:10.352697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:10.352701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:10.352747 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:10.355708 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.356526 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.356531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.356533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.356699 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.357090 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.357093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.357095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.357398 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:10.357613 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:10.357939 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 03:05:10.358128 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260811 03:05:10.364094 2644 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:10.362172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.27.109.1:44345)
0811 03:05:10.362374 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.362378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.362435 (+ 57us) server_negotiation.cc:407] Connection header received
0811 03:05:10.362582 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.362586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.362661 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.362807 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.363764 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36506: BlockingRecv error: recv got EOF from 127.0.0.1:36506 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260811 03:05:10.364455 28084 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786417454123933-28084-0/krb5kdc/krb5cc' of type FILE
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:10.366322 28084 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:10.368000 28084 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:05:10.385728 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:10.369632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.27.109.1:44345)
0811 03:05:10.369818 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.369823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.370256 (+ 433us) server_negotiation.cc:407] Connection header received
0811 03:05:10.370411 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.370415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.370484 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.370585 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.371628 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.372383 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.373852 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.374342 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.374567 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:10.379079 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:10.379085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:10.379089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:10.379134 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:10.382257 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.383442 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.383449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.383453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.383693 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.384187 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.384191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.384193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.384536 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:10.384806 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:10.385228 (+ 422us) server_negotiation.cc:299] Negotiation successful
0811 03:05:10.385443 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 03:05:10.387768 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 2177
I20260811 03:05:10.402508 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:10.388651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.27.109.1:44345)
0811 03:05:10.388826 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.388830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.389370 (+ 540us) server_negotiation.cc:407] Connection header received
0811 03:05:10.389627 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.389632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.389704 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.389861 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.391080 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.391789 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.393129 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.393636 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.393794 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:10.396544 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:10.396550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:10.396554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:10.396593 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:10.399514 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.400328 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.400336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.400344 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.400507 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.400932 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.400936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.400938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.401233 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:10.401535 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:10.401870 (+ 335us) server_negotiation.cc:299] Negotiation successful
0811 03:05:10.402106 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 03:05:10.403216 2322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:10.495657 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 2177
I20260811 03:05:10.518404 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 2334
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:05:10.535745 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:10.519463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.27.109.2:41583)
0811 03:05:10.520052 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.520059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.520087 (+ 28us) server_negotiation.cc:407] Connection header received
0811 03:05:10.520151 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.520158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.520377 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.520526 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.521588 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.522648 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.524063 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.524657 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.524768 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:10.528971 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:10.528991 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:10.529008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:10.529050 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:10.532439 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.533273 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.533281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.533284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.533425 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.533831 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.533839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.533847 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.534255 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:10.534614 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:10.535081 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 03:05:10.535389 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":265,"threads_started":1}
I20260811 03:05:10.537249 2477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:10.628391 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 2334
I20260811 03:05:10.652982 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 2489
Aug 11 03:05:10 dist-test-slave-8tdl krb5kdc[2054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:05:10.671386 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:10.654092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.27.109.3:41999)
0811 03:05:10.654813 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:10.654821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:10.654866 (+ 45us) server_negotiation.cc:407] Connection header received
0811 03:05:10.654934 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:10.654941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:10.655192 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:10.655328 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:10.656525 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.657663 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.659215 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:10.659799 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:10.659910 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:10.664831 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:10.664850 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:10.664866 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:10.664912 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:10.668436 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.669265 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.669274 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.669278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.669439 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:10.669806 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:10.669809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:10.669811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:10.670209 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:10.670488 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:10.670809 (+ 321us) server_negotiation.cc:299] Negotiation successful
0811 03:05:10.671040 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":348,"threads_started":1}
I20260811 03:05:10.673183 2631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:10.761791 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 2489
I20260811 03:05:10.783295 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 2068
I20260811 03:05:10.784529 2143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:10.895077 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 2068
2026-08-11T03:05:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4560 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:05:11 dist-test-slave-8tdl krb5kdc[2665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:05:11 dist-test-slave-8tdl krb5kdc[2665](info): set up 2 sockets
Aug 11 03:05:11 dist-test-slave-8tdl krb5kdc[2665](info): commencing operation
krb5kdc: starting...
W20260811 03:05:12.115577 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:05:12 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:05:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.62.keytab.
Entry for principal oryx/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.62.keytab.
I20260811 03:05:12.423738 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41275
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.62.keytab
--principal=oryx/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:12.798234 2678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:12.798660 2678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:12.798894 2678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:12.807102 2678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:12.807303 2678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:12.807417 2678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:12.807524 2678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:12.807610 2678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:12.818879 2678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.62.keytab
--principal=oryx/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:12.820690 2678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:12.822208 2678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:12.835243 2684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:12.835388 2685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:12.835322 2688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:12.836822 2678 server_base.cc:1034] running on GCE node
I20260811 03:05:12.837754 2678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:12.839459 2678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:12.840699 2678 hybrid_clock.cc:648] HybridClock initialized: now 1786417512840617 us; error 60 us; skew 500 ppm
I20260811 03:05:12.841213 2678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:12 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:12.845175 2678 init.cc:377] Logged in from keytab as oryx/127.27.109.62@KRBTEST.COM (short username oryx)
I20260811 03:05:12.847088 2678 webserver.cc:460] Webserver started at http://127.27.109.62:43517/ using document root <none> and password file <none>
I20260811 03:05:12.847846 2678 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:12.847977 2678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:12.848284 2678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:12.851876 2678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "a3a23560c8fe4b3091ae72834ae3a9cc"
format_stamp: "Formatted at 2026-08-11 03:05:12 on dist-test-slave-8tdl"
server_key: "27c759144372358f5e864255f8f5f0ca"
server_key_iv: "f1b5b6d5f89f418bcd382a6775a57916"
server_key_version: "encryptionkey@0"
I20260811 03:05:12.852821 2678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "a3a23560c8fe4b3091ae72834ae3a9cc"
format_stamp: "Formatted at 2026-08-11 03:05:12 on dist-test-slave-8tdl"
server_key: "27c759144372358f5e864255f8f5f0ca"
server_key_iv: "f1b5b6d5f89f418bcd382a6775a57916"
server_key_version: "encryptionkey@0"
I20260811 03:05:12.859042 2678 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260811 03:05:12.863392 2695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:12.865159 2678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 03:05:12.865414 2678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "a3a23560c8fe4b3091ae72834ae3a9cc"
format_stamp: "Formatted at 2026-08-11 03:05:12 on dist-test-slave-8tdl"
server_key: "27c759144372358f5e864255f8f5f0ca"
server_key_iv: "f1b5b6d5f89f418bcd382a6775a57916"
server_key_version: "encryptionkey@0"
I20260811 03:05:12.865684 2678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:12.890946 2678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:12.896035 2678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:12.896389 2678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:12.919016 2678 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41275
I20260811 03:05:12.919050 2755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41275 every 8 connection(s)
I20260811 03:05:12.920696 2678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:12.924676 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 2678
I20260811 03:05:12.924942 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:12.925184 2756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:12.925809 28084 external_mini_cluster.cc:1428] Setting key 0ded733e69581fa574ac687fd2dfdae0
I20260811 03:05:12.940758 2756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: Bootstrap starting.
Aug 11 03:05:12 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:12.944902 2756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:12.946270 2756 log.cc:824] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:12.949674 2756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: No bootstrap required, opened a new log
I20260811 03:05:12.953699 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:12.928230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.27.109.62:41275)
0811 03:05:12.929221 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:12.929235 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:12.929280 (+ 45us) server_negotiation.cc:407] Connection header received
0811 03:05:12.932067 (+ 2787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:12.932097 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:12.932621 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:12.933041 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:12.935121 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:12.936646 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:12.938550 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:12.939487 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:12.939662 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:12.944918 (+ 5256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:12.944930 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:12.944948 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:12.945056 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:12.948826 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:12.949708 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:12.949715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:12.949723 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:12.949978 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:12.950425 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:12.950432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:12.950436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:12.951121 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:12.951530 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:12.952034 (+ 504us) server_negotiation.cc:299] Negotiation successful
0811 03:05:12.952546 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":491,"threads_started":1}
I20260811 03:05:12.958791 2756 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } }
I20260811 03:05:12.959276 2756 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:12.959390 2756 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a23560c8fe4b3091ae72834ae3a9cc, State: Initialized, Role: FOLLOWER
I20260811 03:05:12.959964 2756 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } }
I20260811 03:05:12.960245 2756 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:12.960397 2756 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:12.960569 2756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:12.962924 2756 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } }
I20260811 03:05:12.963603 2756 leader_election.cc:302] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3a23560c8fe4b3091ae72834ae3a9cc; no voters:
I20260811 03:05:12.964341 2756 leader_election.cc:288] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:12.964457 2763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:12.965324 2763 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 1 LEADER]: Becoming Leader. State: Replica: a3a23560c8fe4b3091ae72834ae3a9cc, State: Running, Role: LEADER
I20260811 03:05:12.966022 2763 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } }
I20260811 03:05:12.967088 2756 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:12.971675 2764 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } }
I20260811 03:05:12.972139 2764 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:12.972659 2765 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3a23560c8fe4b3091ae72834ae3a9cc. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } }
I20260811 03:05:12.973034 2765 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:12.974633 2773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:12.981763 2773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:12.992161 2773 catalog_manager.cc:1347] Generated new cluster ID: 64fdd41ffe424b3887d0b354c4dab236
I20260811 03:05:12.992316 2773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:13.013691 2773 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:13.015703 2773 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:13.026454 2773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: Generated new TSK 0
I20260811 03:05:13.027515 2773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
I20260811 03:05:13.103586 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41275
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:13.477790 2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:13.478147 2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:13.478253 2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:13.485963 2784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:13.486099 2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:13.486217 2784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:05:13.496531 2784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:13.497920 2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:13.499348 2784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:13.514482 2791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:13.514530 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:13.514807 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:13.515779 2784 server_base.cc:1034] running on GCE node
I20260811 03:05:13.516489 2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:13.517598 2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:13.518847 2784 hybrid_clock.cc:648] HybridClock initialized: now 1786417513518794 us; error 39 us; skew 500 ppm
I20260811 03:05:13.519203 2784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:13 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417513, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:13.522799 2784 init.cc:377] Logged in from keytab as oryx/127.27.109.1@KRBTEST.COM (short username oryx)
I20260811 03:05:13.524325 2784 webserver.cc:460] Webserver started at http://127.27.109.1:35907/ using document root <none> and password file <none>
I20260811 03:05:13.525041 2784 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:13.525179 2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:13.525521 2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:13.528751 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "6552402ca2054fbcb005b9ce1f80c2e5"
format_stamp: "Formatted at 2026-08-11 03:05:13 on dist-test-slave-8tdl"
server_key: "c6e9c37b68e94fb96885a42a223b73d6"
server_key_iv: "a609784202bdb471b4aeaaf4a4c47588"
server_key_version: "encryptionkey@0"
I20260811 03:05:13.529618 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "6552402ca2054fbcb005b9ce1f80c2e5"
format_stamp: "Formatted at 2026-08-11 03:05:13 on dist-test-slave-8tdl"
server_key: "c6e9c37b68e94fb96885a42a223b73d6"
server_key_iv: "a609784202bdb471b4aeaaf4a4c47588"
server_key_version: "encryptionkey@0"
I20260811 03:05:13.535341 2784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 03:05:13.539261 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:13.540841 2784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:05:13.541069 2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "6552402ca2054fbcb005b9ce1f80c2e5"
format_stamp: "Formatted at 2026-08-11 03:05:13 on dist-test-slave-8tdl"
server_key: "c6e9c37b68e94fb96885a42a223b73d6"
server_key_iv: "a609784202bdb471b4aeaaf4a4c47588"
server_key_version: "encryptionkey@0"
I20260811 03:05:13.541322 2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:13.561508 2784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:13.566239 2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:13.566578 2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:13.567804 2784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:13.569288 2784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:13.569367 2784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:13.569535 2784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:13.569682 2784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260811 03:05:13.606971 2784 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:39493
I20260811 03:05:13.607089 2924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:39493 every 8 connection(s)
I20260811 03:05:13.608424 2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:05:13.613441 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 2784
I20260811 03:05:13.613714 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:05:13.614424 28084 external_mini_cluster.cc:1428] Setting key ecc3e95142c3659342af8e00081159fc
Aug 11 03:05:13 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417513, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:13.632077 2927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:13.611492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:40529 (local address 127.27.109.62:41275)
0811 03:05:13.611984 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:13.611988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:13.612942 (+ 954us) server_negotiation.cc:407] Connection header received
0811 03:05:13.615076 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:13.615083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:13.615181 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:13.615303 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:13.617245 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:13.617988 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:13.619737 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:13.620149 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:13.620292 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:13.624488 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:13.624499 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:13.624506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:13.624568 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:13.627989 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:13.629181 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:13.629186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:13.629188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:13.629319 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:13.629901 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:13.629905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:13.629907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:13.630216 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:13.630341 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:13.631593 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0811 03:05:13.631800 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":268,"threads_started":1}
I20260811 03:05:13.634011 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:13.611998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.1:40529)
0811 03:05:13.612751 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:13.612791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:13.614757 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:13.615602 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:13.615615 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:13.616416 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:13.617055 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:13.617075 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:13.618172 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:13.618176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:13.619564 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:13.619578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:13.620414 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:13.620422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:13.620658 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:13.621397 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:13.621429 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:13.624169 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:13.628190 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:13.628205 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:13.628230 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:13.629019 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:13.629447 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:13.629455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:13.629463 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:13.629758 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:13.630364 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:13.630376 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:13.631024 (+ 648us) client_negotiation.cc:769] Sending connection context
0811 03:05:13.631474 (+ 450us) client_negotiation.cc:240] Negotiation successful
0811 03:05:13.632193 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":293,"threads_started":1}
I20260811 03:05:13.635972 2925 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41275
I20260811 03:05:13.636363 2925 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:13.637213 2925 heartbeater.cc:511] Master 127.27.109.62:41275 requested a full tablet report, sending...
I20260811 03:05:13.639665 2715 ts_manager.cc:194] Registered new tserver with Master: 6552402ca2054fbcb005b9ce1f80c2e5 (127.27.109.1:39493)
I20260811 03:05:13.641286 2715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:40529
WARNING: no policy specified for oryx/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
I20260811 03:05:13.687507 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41275
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:14.056808 2932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:14.057121 2932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:14.057301 2932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:14.065073 2932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:14.065203 2932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:14.065387 2932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:14.076282 2932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:14.077755 2932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:14.079291 2932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:14.093822 2937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:14.093933 2938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:14.093781 2940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:14.095516 2932 server_base.cc:1034] running on GCE node
I20260811 03:05:14.096297 2932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:14.097414 2932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:14.098724 2932 hybrid_clock.cc:648] HybridClock initialized: now 1786417514098606 us; error 65 us; skew 500 ppm
I20260811 03:05:14.099108 2932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:14 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417514, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:14.103017 2932 init.cc:377] Logged in from keytab as oryx/127.27.109.2@KRBTEST.COM (short username oryx)
I20260811 03:05:14.104852 2932 webserver.cc:460] Webserver started at http://127.27.109.2:42887/ using document root <none> and password file <none>
I20260811 03:05:14.105602 2932 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:14.105712 2932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:14.106029 2932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:14.109450 2932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "863bd409cd0d4058bdc16227e559e379"
format_stamp: "Formatted at 2026-08-11 03:05:14 on dist-test-slave-8tdl"
server_key: "b869650bc8d012645ce2ccf87cb00cce"
server_key_iv: "c13b5f614c31c551a7ab2d1178abdb38"
server_key_version: "encryptionkey@0"
I20260811 03:05:14.110364 2932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "863bd409cd0d4058bdc16227e559e379"
format_stamp: "Formatted at 2026-08-11 03:05:14 on dist-test-slave-8tdl"
server_key: "b869650bc8d012645ce2ccf87cb00cce"
server_key_iv: "c13b5f614c31c551a7ab2d1178abdb38"
server_key_version: "encryptionkey@0"
I20260811 03:05:14.116447 2932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260811 03:05:14.120461 2947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:14.122152 2932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 03:05:14.122313 2932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "863bd409cd0d4058bdc16227e559e379"
format_stamp: "Formatted at 2026-08-11 03:05:14 on dist-test-slave-8tdl"
server_key: "b869650bc8d012645ce2ccf87cb00cce"
server_key_iv: "c13b5f614c31c551a7ab2d1178abdb38"
server_key_version: "encryptionkey@0"
I20260811 03:05:14.122506 2932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:14.140477 2932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:14.145658 2932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:14.145959 2932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:14.147308 2932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:14.148859 2932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:14.148968 2932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:14.149073 2932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:14.149118 2932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:14.186292 2932 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45191
I20260811 03:05:14.186338 3074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45191 every 8 connection(s)
I20260811 03:05:14.187738 2932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:14.197782 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 2932
I20260811 03:05:14.198019 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:05:14.198642 28084 external_mini_cluster.cc:1428] Setting key 92434f21e2fa384e76c8e6d2569a26e4
Aug 11 03:05:14 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417514, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:14.213441 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:14.190774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:44523 (local address 127.27.109.62:41275)
0811 03:05:14.191308 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:14.191315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:14.192359 (+ 1044us) server_negotiation.cc:407] Connection header received
0811 03:05:14.194248 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:14.194253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:14.194329 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:14.194435 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:14.196383 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:14.197149 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:14.199742 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:14.200419 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:14.200554 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:14.205488 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:14.205495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:14.205498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:14.205543 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:14.208742 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:14.209927 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:14.209933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:14.209935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:14.210185 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:14.210804 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:14.210808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:14.210810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:14.211242 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:14.211387 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:14.212533 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0811 03:05:14.212850 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":270,"threads_started":1}
I20260811 03:05:14.214248 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:14.191355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.2:44523)
0811 03:05:14.192182 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:14.192222 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:14.193931 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:14.194868 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:14.194879 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:14.195506 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:14.196162 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:14.196182 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:14.197406 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:14.197412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:14.199558 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:14.199573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:14.200687 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:14.200694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:14.201134 (+ 440us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:14.201967 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:14.201996 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:14.205121 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:14.208948 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:14.208962 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:14.208987 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:14.209740 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:14.210334 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:14.210342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:14.210348 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:14.210652 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:14.211415 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:14.211427 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:14.211970 (+ 543us) client_negotiation.cc:769] Sending connection context
0811 03:05:14.212340 (+ 370us) client_negotiation.cc:240] Negotiation successful
0811 03:05:14.213034 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":297,"threads_started":1}
I20260811 03:05:14.216135 3075 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41275
I20260811 03:05:14.216501 3075 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:14.217306 3075 heartbeater.cc:511] Master 127.27.109.62:41275 requested a full tablet report, sending...
I20260811 03:05:14.219038 2715 ts_manager.cc:194] Registered new tserver with Master: 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191)
I20260811 03:05:14.219905 2715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:44523
WARNING: no policy specified for oryx/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
I20260811 03:05:14.262885 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41275
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:14.638866 3081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:14.639230 3081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:14.639457 3081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 03:05:14.644929 2925 heartbeater.cc:503] Master 127.27.109.62:41275 was elected leader, sending a full tablet report...
W20260811 03:05:14.648761 3081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:14.648908 3081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:14.649041 3081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:14.660663 3081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:14.662098 3081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:14.663540 3081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:14.678854 3086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:14.678982 3087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:14.679036 3090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:14.680236 3081 server_base.cc:1034] running on GCE node
I20260811 03:05:14.681111 3081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:14.682325 3081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:14.683578 3081 hybrid_clock.cc:648] HybridClock initialized: now 1786417514683522 us; error 71 us; skew 500 ppm
I20260811 03:05:14.683947 3081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:14 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417514, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:14.687819 3081 init.cc:377] Logged in from keytab as oryx/127.27.109.3@KRBTEST.COM (short username oryx)
I20260811 03:05:14.689807 3081 webserver.cc:460] Webserver started at http://127.27.109.3:34557/ using document root <none> and password file <none>
I20260811 03:05:14.690629 3081 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:14.690851 3081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:14.691236 3081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:14.694444 3081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "39af5b9dd1f44f419bb5374224bff222"
format_stamp: "Formatted at 2026-08-11 03:05:14 on dist-test-slave-8tdl"
server_key: "1eee159709aff5390b135889e0ce4443"
server_key_iv: "6b19bc5c77e65b220ff3892fa121dc13"
server_key_version: "encryptionkey@0"
I20260811 03:05:14.695483 3081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "39af5b9dd1f44f419bb5374224bff222"
format_stamp: "Formatted at 2026-08-11 03:05:14 on dist-test-slave-8tdl"
server_key: "1eee159709aff5390b135889e0ce4443"
server_key_iv: "6b19bc5c77e65b220ff3892fa121dc13"
server_key_version: "encryptionkey@0"
I20260811 03:05:14.701216 3081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260811 03:05:14.705150 3097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:14.707013 3081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:14.707182 3081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "39af5b9dd1f44f419bb5374224bff222"
format_stamp: "Formatted at 2026-08-11 03:05:14 on dist-test-slave-8tdl"
server_key: "1eee159709aff5390b135889e0ce4443"
server_key_iv: "6b19bc5c77e65b220ff3892fa121dc13"
server_key_version: "encryptionkey@0"
I20260811 03:05:14.707382 3081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:14.728401 3081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:14.733320 3081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:14.733587 3081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:14.734582 3081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:14.736104 3081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:14.736187 3081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:14.736277 3081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:14.736322 3081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:14.772998 3081 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:40047
I20260811 03:05:14.773037 3226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:40047 every 8 connection(s)
I20260811 03:05:14.774394 3081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:14.783556 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 3081
I20260811 03:05:14.783808 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:05:14.784461 28084 external_mini_cluster.cc:1428] Setting key 34c43fbd2385df13213972a3cae46e69
Aug 11 03:05:14 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417514, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:14.797823 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:14.777418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:33745 (local address 127.27.109.62:41275)
0811 03:05:14.777980 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:14.777985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:14.779200 (+ 1215us) server_negotiation.cc:407] Connection header received
0811 03:05:14.781338 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:14.781344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:14.781464 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:14.781593 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:14.783473 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:14.784793 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:14.786885 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:14.787578 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:14.787725 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:14.791761 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:14.791767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:14.791770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:14.791809 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:14.794600 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:14.795522 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:14.795527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:14.795529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:14.795661 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:14.796155 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:14.796159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:14.796161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:14.796453 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:14.796581 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:14.797346 (+ 765us) server_negotiation.cc:299] Negotiation successful
0811 03:05:14.797534 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":256,"threads_started":1}
I20260811 03:05:14.799098 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:14.778057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.3:33745)
0811 03:05:14.779005 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:14.779050 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:14.781018 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:14.781888 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:14.781897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:14.782495 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:14.783196 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:14.783242 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:14.784976 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:14.784982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:14.786718 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:14.786735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:14.787895 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:14.787902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:14.788131 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:14.788899 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:14.788931 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:14.791476 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:14.794835 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:14.794853 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:14.794869 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:14.795372 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:14.795797 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:14.795800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:14.795803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:14.796015 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:14.796591 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:14.796597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:14.796950 (+ 353us) client_negotiation.cc:769] Sending connection context
0811 03:05:14.797229 (+ 279us) client_negotiation.cc:240] Negotiation successful
0811 03:05:14.797707 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":381,"threads_started":1}
I20260811 03:05:14.800697 3227 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41275
I20260811 03:05:14.801064 3227 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:14.801832 3227 heartbeater.cc:511] Master 127.27.109.62:41275 requested a full tablet report, sending...
I20260811 03:05:14.803366 2715 ts_manager.cc:194] Registered new tserver with Master: 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047)
I20260811 03:05:14.804268 2715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:33745
I20260811 03:05:14.814019 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.61.keytab.
Entry for principal oryx/127.27.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.61.keytab.
I20260811 03:05:14.879818 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:32823
--webserver_interface=127.27.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.61.keytab
--principal=oryx/127.27.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41275,127.27.109.61:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:05:15.222805 3075 heartbeater.cc:503] Master 127.27.109.62:41275 was elected leader, sending a full tablet report...
W20260811 03:05:15.249248 3234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:15.249655 3234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:15.249825 3234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:15.257691 3234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:15.257864 3234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:15.257982 3234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:15.258090 3234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:15.258168 3234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:15.269062 3234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41275,127.27.109.61:32823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.61.keytab
--principal=oryx/127.27.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
W20260811 03:05:15.269541 3234 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.109.62:41275,127.27.109.61:32823'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260811 03:05:15.270773 3234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:15.272466 3234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:15.286021 3239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:15.286132 3242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:15.286195 3240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:15.292469 3234 server_base.cc:1034] running on GCE node
I20260811 03:05:15.293422 3234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:15.295092 3234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:15.296399 3234 hybrid_clock.cc:648] HybridClock initialized: now 1786417515296307 us; error 44 us; skew 500 ppm
I20260811 03:05:15.296860 3234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:15 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417515, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:15.301373 3234 init.cc:377] Logged in from keytab as oryx/127.27.109.61@KRBTEST.COM (short username oryx)
I20260811 03:05:15.303287 3234 webserver.cc:460] Webserver started at http://127.27.109.61:38591/ using document root <none> and password file <none>
I20260811 03:05:15.304122 3234 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:15.304251 3234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:15.304643 3234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:15.308399 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data/instance:
uuid: "00d96f2bcd114aa6a24abaf4352cd7c7"
format_stamp: "Formatted at 2026-08-11 03:05:15 on dist-test-slave-8tdl"
server_key: "5b4830d5bed97969ce94487f37077e25"
server_key_iv: "46fc7f7b4b3dff57811a762275bf9d59"
server_key_version: "encryptionkey@0"
I20260811 03:05:15.309346 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal/instance:
uuid: "00d96f2bcd114aa6a24abaf4352cd7c7"
format_stamp: "Formatted at 2026-08-11 03:05:15 on dist-test-slave-8tdl"
server_key: "5b4830d5bed97969ce94487f37077e25"
server_key_iv: "46fc7f7b4b3dff57811a762275bf9d59"
server_key_version: "encryptionkey@0"
I20260811 03:05:15.315893 3234 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260811 03:05:15.320281 3250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:15.322063 3234 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:05:15.322290 3234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
uuid: "00d96f2bcd114aa6a24abaf4352cd7c7"
format_stamp: "Formatted at 2026-08-11 03:05:15 on dist-test-slave-8tdl"
server_key: "5b4830d5bed97969ce94487f37077e25"
server_key_iv: "46fc7f7b4b3dff57811a762275bf9d59"
server_key_version: "encryptionkey@0"
I20260811 03:05:15.322554 3234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:15.337827 3234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:15.342859 3234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:15.343192 3234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:15.365085 3234 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:32823
I20260811 03:05:15.365149 3310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:32823 every 8 connection(s)
I20260811 03:05:15.366710 3234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
I20260811 03:05:15.369500 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 3234
I20260811 03:05:15.372126 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:15.382606 3311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:05:15 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
Aug 11 03:05:15 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417515, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.61@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:15.399215 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.372874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.27.109.61:32823)
0811 03:05:15.373793 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:15.373806 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:15.373853 (+ 47us) server_negotiation.cc:407] Connection header received
0811 03:05:15.375806 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:15.375834 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:15.376306 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:15.376722 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:15.378842 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.380577 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.382318 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.383278 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.383443 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.388873 (+ 5430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:15.388885 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:15.388902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:15.388957 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:15.393002 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.394013 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.394023 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.394033 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.394321 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.394953 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.394959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.394963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.395866 (+ 903us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:15.396154 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:15.396858 (+ 704us) server_negotiation.cc:299] Negotiation successful
0811 03:05:15.397441 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":404,"threads_started":1}
I20260811 03:05:15.406237 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.386117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:58914)
0811 03:05:15.387110 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:15.387155 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:15.387353 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:15.387785 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:15.387792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:15.388453 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:15.388778 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.388791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.390130 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.390137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.392168 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.392186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.393054 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.393064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.393242 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.393947 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:15.394055 (+ 108us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:15.397356 (+ 3301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:15.403255 (+ 5899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.403266 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.403273 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.403789 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.404301 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.404305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.404308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.404490 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.405186 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:15.405194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:15.405404 (+ 210us) client_negotiation.cc:769] Sending connection context
0811 03:05:15.405586 (+ 182us) client_negotiation.cc:240] Negotiation successful
0811 03:05:15.405903 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":500,"threads_started":1}
I20260811 03:05:15.406453 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.386040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.27.109.62:41275)
0811 03:05:15.386741 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:15.386748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:15.387255 (+ 507us) server_negotiation.cc:407] Connection header received
0811 03:05:15.387515 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:15.387519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:15.387594 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:15.387716 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:15.388995 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.389956 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.392358 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.392815 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.392923 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.397611 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:15.397620 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:15.397625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:15.397691 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:15.403085 (+ 5394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.403959 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.403966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.403970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.404155 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.404629 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.404634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.404636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.405034 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:15.405182 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:15.405770 (+ 588us) server_negotiation.cc:299] Negotiation successful
0811 03:05:15.406144 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":302,"threads_started":1}
I20260811 03:05:15.408741 3311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:05:15 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417515, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.61@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260811 03:05:15.423671 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.410101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.0.0.1:51544)
0811 03:05:15.410294 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:15.410319 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:15.410480 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:15.410882 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:15.410886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:15.411409 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:15.411561 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.411570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.412745 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.412751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.413678 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.413691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.414439 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.414444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.414639 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.415333 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:15.415352 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:15.417840 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:15.421018 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.421025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.421030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.421451 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.421850 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.421855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.421858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.421986 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.422519 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:15.422524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:15.422716 (+ 192us) client_negotiation.cc:769] Sending connection context
0811 03:05:15.422882 (+ 166us) client_negotiation.cc:240] Negotiation successful
0811 03:05:15.423207 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 03:05:15.423730 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.409992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.27.109.61:32823)
0811 03:05:15.410249 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:15.410260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:15.410402 (+ 142us) server_negotiation.cc:407] Connection header received
0811 03:05:15.410597 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:15.410608 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:15.410669 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:15.410842 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:15.411718 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.412590 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.413844 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.414274 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.414415 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.418110 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:15.418116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:15.418119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:15.418151 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:15.420814 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.421593 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.421600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.421608 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.421730 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.422092 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.422098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.422101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.422404 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:15.422557 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:15.422977 (+ 420us) server_negotiation.cc:299] Negotiation successful
0811 03:05:15.423368 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":81}
I20260811 03:05:15.430019 3311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Bootstrap starting.
I20260811 03:05:15.433640 3311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:15.434935 3311 log.cc:824] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:15.438148 3311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: No bootstrap required, opened a new log
I20260811 03:05:15.448179 3311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } }
I20260811 03:05:15.448603 3311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:15.448693 3311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d96f2bcd114aa6a24abaf4352cd7c7, State: Initialized, Role: FOLLOWER
I20260811 03:05:15.449312 3311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } }
I20260811 03:05:15.450469 3318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } } }
I20260811 03:05:15.451011 3318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:05:15.451649 3311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 03:05:15.462832 3329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:05:15.463033 3329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:05:15.465909 3234 master_runner.cc:428] Detected that this master 00d96f2bcd114aa6a24abaf4352cd7c7 is joining an existing cluster
I20260811 03:05:15.466046 3234 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.109.61:32823
I20260811 03:05:15.487735 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.470721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.27.109.61:32823)
0811 03:05:15.470973 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:15.470983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:15.472732 (+ 1749us) server_negotiation.cc:407] Connection header received
0811 03:05:15.472883 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:15.472888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:15.472992 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:15.473126 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:15.474473 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.475686 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.477767 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.478578 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.478743 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.481171 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:15.481176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:15.481179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:15.481206 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:15.484300 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.485069 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.485077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.485083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.485290 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.485777 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.485783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.485788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.486292 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:15.486538 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:15.486935 (+ 397us) server_negotiation.cc:299] Negotiation successful
0811 03:05:15.487320 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":186,"server-negotiator.queue_time_us":78,"spinlock_wait_cycles":22400}
I20260811 03:05:15.487761 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.470790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.0.0.1:51546)
0811 03:05:15.472621 (+ 1831us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:15.472640 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:15.472751 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:15.473189 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:15.473197 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:15.473922 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:15.474319 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.474333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.475890 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.475897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.477610 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.477623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.478828 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.478835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.479055 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.480093 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:15.480143 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:15.480968 (+ 825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:15.484451 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.484457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.484461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.484923 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.485428 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.485436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.485445 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.485658 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.486415 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:15.486425 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:15.486632 (+ 207us) client_negotiation.cc:769] Sending connection context
0811 03:05:15.486842 (+ 210us) client_negotiation.cc:240] Negotiation successful
0811 03:05:15.487196 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1688,"mutex_wait_us":1,"thread_start_us":1423,"threads_started":1}
I20260811 03:05:15.488484 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.471519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.27.109.62:41275)
0811 03:05:15.471728 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:15.471736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:15.471829 (+ 93us) server_negotiation.cc:407] Connection header received
0811 03:05:15.472157 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:15.472166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:15.472281 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:15.472367 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:15.474403 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.475270 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.477627 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.478178 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.478332 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.481169 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:15.481176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:15.481181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:15.481228 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:15.484985 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.485724 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.485732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.485736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.485934 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.486528 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.486534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.486537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.487030 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:15.487183 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:15.487870 (+ 687us) server_negotiation.cc:299] Negotiation successful
0811 03:05:15.488127 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 03:05:15.489701 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.470741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:58918)
0811 03:05:15.471675 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:15.471698 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:15.471962 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:15.472652 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:15.472660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:15.473685 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:15.474158 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.474173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.475443 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.475447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.477365 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.477403 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.478417 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.478426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.478632 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.479504 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:15.479528 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:15.480913 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:15.485119 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.485126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.485135 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.485588 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.486173 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.486178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.486182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.486368 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.487290 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:15.487295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:15.487516 (+ 221us) client_negotiation.cc:769] Sending connection context
0811 03:05:15.488817 (+ 1301us) client_negotiation.cc:240] Negotiation successful
0811 03:05:15.489193 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":40,"thread_start_us":1009,"threads_started":1}
Aug 11 03:05:15 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260811 03:05:15.517009 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.497354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.27.109.61:32823)
0811 03:05:15.497573 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:15.497584 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:15.498258 (+ 674us) server_negotiation.cc:407] Connection header received
0811 03:05:15.498592 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:15.498598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:15.498780 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:15.498913 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:15.500239 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.501429 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.503118 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.503864 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.504019 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.508803 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:15.508811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:15.508816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:15.508863 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:15.513163 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.514266 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.514273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.514277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.514474 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.515038 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.515044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.515048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.515509 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:15.515680 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:15.516318 (+ 638us) server_negotiation.cc:299] Negotiation successful
0811 03:05:15.516631 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 03:05:15.517167 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.497318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.0.0.1:51552)
0811 03:05:15.498092 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:15.498142 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:15.498339 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:15.498989 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:15.498998 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:15.499784 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:15.500059 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.500070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.501622 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.501626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.502933 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.502948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.504214 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.504227 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.504501 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.505334 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:15.505366 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:15.508553 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:15.513349 (+ 4796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.513372 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.513389 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.514101 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.514619 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.514626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.514633 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.514897 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.515673 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:15.515685 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:15.515946 (+ 261us) client_negotiation.cc:769] Sending connection context
0811 03:05:15.516171 (+ 225us) client_negotiation.cc:240] Negotiation successful
0811 03:05:15.516609 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":344,"threads_started":1}
I20260811 03:05:15.519702 2715 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.109.61:32823: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7"
member_type: NON_VOTER
last_known_addr {
host: "127.27.109.61"
port: 32823
}
attrs {
promote: true
}
}
dest_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc"
cas_config_opid_index: -1
I20260811 03:05:15.520872 2715 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } }
I20260811 03:05:15.532733 3282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:15.534825 3341 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 00d96f2bcd114aa6a24abaf4352cd7c7 (127.27.109.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } }
I20260811 03:05:15.536916 3282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Refusing update from remote peer a3a23560c8fe4b3091ae72834ae3a9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260811 03:05:15.540956 3345 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:15.541710 3341 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.109.61:32823
I20260811 03:05:15.542603 3341 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } }
I20260811 03:05:15.542884 3234 master.cc:537] Master@127.27.109.61:32823 shutting down...
I20260811 03:05:15.543040 3341 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.27.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.60@KRBTEST.COM" created.
I20260811 03:05:15.560431 3234 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:15.560973 3234 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:15.561228 3234 tablet_replica.cc:330] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: stopping tablet replica
W20260811 03:05:15.571686 2697 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260811 03:05:15.580793 3234 master.cc:552] Master@127.27.109.61:32823 shutdown complete.
I20260811 03:05:15.581144 3234 master_runner.cc:305] Clearing existing system tablet
I20260811 03:05:15.581950 3234 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:15.582190 3234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260811 03:05:15.583770 2697 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc -> Peer 00d96f2bcd114aa6a24abaf4352cd7c7 (127.27.109.61:32823): Couldn't send request to peer 00d96f2bcd114aa6a24abaf4352cd7c7. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260811 03:05:15.585794 3234 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260811 03:05:15.587883 3351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:15.588387 3234 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:15.588792 3234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 03:05:15.588902 3234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
uuid: "00d96f2bcd114aa6a24abaf4352cd7c7"
format_stamp: "Formatted at 2026-08-11 03:05:15 on dist-test-slave-8tdl"
server_key: "5b4830d5bed97969ce94487f37077e25"
server_key_iv: "46fc7f7b4b3dff57811a762275bf9d59"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.60.keytab.
Entry for principal oryx/127.27.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.60.keytab.
I20260811 03:05:15.593528 3234 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:15.595705 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:42879
--webserver_interface=127.27.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.60.keytab
--principal=oryx/127.27.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41275,127.27.109.61:32823,127.27.109.60:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:05:15.602890 3234 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 03:05:15.603175 3234 log.cc:1196] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260811 03:05:15.603765 3234 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Deleting consensus metadata
I20260811 03:05:15.604156 3234 master_runner.cc:315] Copying system tablet from 127.27.109.62:41275
I20260811 03:05:15.613071 3234 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: tablet copy: Beginning tablet copy session from remote peer at address 127.27.109.62:41275
I20260811 03:05:15.628991 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.614031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.27.109.62:41275)
0811 03:05:15.614290 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:15.614300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:15.614938 (+ 638us) server_negotiation.cc:407] Connection header received
0811 03:05:15.615170 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:15.615176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:15.615291 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:15.615427 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:15.616450 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.617284 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.618881 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.619387 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.619548 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.621819 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:15.621825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:15.621829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:15.621868 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:15.625165 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.626084 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.626091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.626102 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.626332 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:15.627100 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:15.627106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:15.627110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:15.627596 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:15.627853 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:15.628282 (+ 429us) server_negotiation.cc:299] Negotiation successful
0811 03:05:15.628543 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:05:15.629070 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:15.614013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:58926)
0811 03:05:15.614783 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:15.614806 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:15.615008 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:15.615517 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:15.615522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:15.616034 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:15.616276 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.616286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.617536 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.617547 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.618629 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:15.618644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:15.619596 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:15.619603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:15.619821 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:15.620602 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:15.620623 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:15.621565 (+ 942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:15.625364 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.625371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.625376 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.625921 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.626529 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:15.626535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:15.626544 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:15.626935 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:15.627747 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:15.627753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:15.627982 (+ 229us) client_negotiation.cc:769] Sending connection context
0811 03:05:15.628147 (+ 165us) client_negotiation.cc:240] Negotiation successful
0811 03:05:15.628545 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":446,"threads_started":1}
I20260811 03:05:15.630272 2738 tablet_copy_service.cc:140] P a3a23560c8fe4b3091ae72834ae3a9cc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 00d96f2bcd114aa6a24abaf4352cd7c7 ({username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:58926)
I20260811 03:05:15.630584 2738 tablet_copy_service.cc:161] P a3a23560c8fe4b3091ae72834ae3a9cc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 00d96f2bcd114aa6a24abaf4352cd7c7 at {username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:58926: session id = 00d96f2bcd114aa6a24abaf4352cd7c7-00000000000000000000000000000000
I20260811 03:05:15.633970 2738 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 03:05:15.636363 3234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 03:05:15.641479 3234 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: tablet copy: Starting download of 0 data blocks...
I20260811 03:05:15.641849 3234 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: tablet copy: Starting download of 1 WAL segments...
I20260811 03:05:15.644853 3234 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 03:05:15.649663 2738 tablet_copy_service.cc:342] P a3a23560c8fe4b3091ae72834ae3a9cc: Request end of tablet copy session 00d96f2bcd114aa6a24abaf4352cd7c7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:58926
I20260811 03:05:15.649911 2738 tablet_copy_service.cc:433] P a3a23560c8fe4b3091ae72834ae3a9cc: ending tablet copy session 00d96f2bcd114aa6a24abaf4352cd7c7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 00d96f2bcd114aa6a24abaf4352cd7c7
I20260811 03:05:15.664201 3234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:15.667626 3361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:15.678468 3364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:15.679039 3362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:15.680917 3234 server_base.cc:1034] running on GCE node
I20260811 03:05:15.681411 3234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:15.682265 3234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:15.683624 3234 hybrid_clock.cc:648] HybridClock initialized: now 1786417515683539 us; error 38 us; skew 500 ppm
I20260811 03:05:15.683959 3234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:15 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417515, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:15.686468 3234 init.cc:377] Logged in from keytab as oryx/127.27.109.61@KRBTEST.COM (short username oryx)
I20260811 03:05:15.691680 3234 webserver.cc:460] Webserver started at http://127.27.109.61:44421/ using document root <none> and password file <none>
I20260811 03:05:15.692126 3234 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:15.692221 3234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:15.695740 3234 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 03:05:15.697969 3374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:15.698480 3234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 03:05:15.698732 3234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
uuid: "00d96f2bcd114aa6a24abaf4352cd7c7"
format_stamp: "Formatted at 2026-08-11 03:05:15 on dist-test-slave-8tdl"
server_key: "5b4830d5bed97969ce94487f37077e25"
server_key_iv: "46fc7f7b4b3dff57811a762275bf9d59"
server_key_version: "encryptionkey@0"
I20260811 03:05:15.699015 3234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:15.717088 3234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:15.724259 3234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:15.798532 3234 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:32823
I20260811 03:05:15.798556 3427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:32823 every 8 connection(s)
I20260811 03:05:15.799813 3234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
I20260811 03:05:15.807420 3227 heartbeater.cc:503] Master 127.27.109.62:41275 was elected leader, sending a full tablet report...
I20260811 03:05:15.811455 3428 sys_catalog.cc:263] Verifying existing consensus state
I20260811 03:05:15.815976 3428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Bootstrap starting.
I20260811 03:05:15.838464 3428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 03:05:15.839160 3428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Bootstrap complete.
I20260811 03:05:15.840399 3428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } }
I20260811 03:05:15.840723 3428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 00d96f2bcd114aa6a24abaf4352cd7c7, State: Initialized, Role: LEARNER
I20260811 03:05:15.841046 3428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } }
I20260811 03:05:15.842026 3431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } }
I20260811 03:05:15.842341 3428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:15.842538 3431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: This master's current role is: LEARNER
I20260811 03:05:15.869871 3443 catalog_manager.cc:1259] Loaded cluster ID: 64fdd41ffe424b3887d0b354c4dab236
I20260811 03:05:15.870304 3443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: loading cluster ID for follower catalog manager: success
I20260811 03:05:15.874423 3443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: acquiring CA information for follower catalog manager: success
I20260811 03:05:15.877547 3443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 03:05:16.068307 3353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:16.068657 3353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:16.068830 3353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:16.077004 3353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:16.077136 3353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:16.077229 3353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:16.077323 3353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:16.077414 3353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:16.088505 3353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41275,127.27.109.61:32823,127.27.109.60:42879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.60.keytab
--principal=oryx/127.27.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:16.090027 3353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:16.091589 3353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:16.104858 3449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:16.104939 3451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:16.105082 3448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:16.107004 3353 server_base.cc:1034] running on GCE node
I20260811 03:05:16.108392 3353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:16.111111 3353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:16.112501 3353 hybrid_clock.cc:648] HybridClock initialized: now 1786417516112428 us; error 54 us; skew 500 ppm
I20260811 03:05:16.113190 3353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:16 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417516, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:16.118602 3353 init.cc:377] Logged in from keytab as oryx/127.27.109.60@KRBTEST.COM (short username oryx)
I20260811 03:05:16.121157 3353 webserver.cc:460] Webserver started at http://127.27.109.60:45985/ using document root <none> and password file <none>
I20260811 03:05:16.122042 3353 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:16.122201 3353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:16.122591 3353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:16.125587 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.106278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.0.0.1:51566)
0811 03:05:16.106768 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.106818 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.106987 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.119150 (+ 12163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.119157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.119202 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:16.119436 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.119447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.121543 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.121548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.123183 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.123209 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.124926 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.124931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.125102 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.125150 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.125235 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.125345 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":198,"threads_started":1}
I20260811 03:05:16.125821 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.106518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.27.109.61:32823)
0811 03:05:16.118383 (+ 11865us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.118390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.118417 (+ 27us) server_negotiation.cc:407] Connection header received
0811 03:05:16.118582 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.118595 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.118760 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.119053 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:16.119648 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.121239 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.123405 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.124722 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.124889 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.125125 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.125298 (+ 173us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.125453 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11689,"thread_start_us":11543,"threads_started":1}
I20260811 03:05:16.127090 3403 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260811 03:05:16.127193 3353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data/instance:
uuid: "925210eabbaa41edb648ac0089514df7"
format_stamp: "Formatted at 2026-08-11 03:05:16 on dist-test-slave-8tdl"
server_key: "36abe30533b9b98d38c790cff6e7a229"
server_key_iv: "680efbd916dd1a4aa23fff2e82e4f9ee"
server_key_version: "encryptionkey@0"
I20260811 03:05:16.127754 3431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3a23560c8fe4b3091ae72834ae3a9cc. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } }
I20260811 03:05:16.128028 3431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: This master's current role is: LEARNER
I20260811 03:05:16.128305 3353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal/instance:
uuid: "925210eabbaa41edb648ac0089514df7"
format_stamp: "Formatted at 2026-08-11 03:05:16 on dist-test-slave-8tdl"
server_key: "36abe30533b9b98d38c790cff6e7a229"
server_key_iv: "680efbd916dd1a4aa23fff2e82e4f9ee"
server_key_version: "encryptionkey@0"
I20260811 03:05:16.134186 3353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:05:16.138227 3461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:16.139840 3353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:16.140146 3353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
uuid: "925210eabbaa41edb648ac0089514df7"
format_stamp: "Formatted at 2026-08-11 03:05:16 on dist-test-slave-8tdl"
server_key: "36abe30533b9b98d38c790cff6e7a229"
server_key_iv: "680efbd916dd1a4aa23fff2e82e4f9ee"
server_key_version: "encryptionkey@0"
I20260811 03:05:16.140384 3353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:16.166546 3353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:16.172098 3353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:16.172431 3353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:16.193832 3353 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:42879
I20260811 03:05:16.193835 3517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:42879 every 8 connection(s)
I20260811 03:05:16.195477 3353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
I20260811 03:05:16.199568 3518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:16.205194 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 3353
I20260811 03:05:16.206845 3518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:05:16 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417516, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
Aug 11 03:05:16 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
I20260811 03:05:16.227818 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.210224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.27.109.62:41275)
0811 03:05:16.210799 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.210805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.211378 (+ 573us) server_negotiation.cc:407] Connection header received
0811 03:05:16.211913 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.211920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.212013 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.212213 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.213538 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.214324 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.216003 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.216446 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.216586 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.220777 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.220783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.220787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.220825 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.224014 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.224901 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.224906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.224909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.225059 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.225660 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.225665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.225668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.226028 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.226181 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.227194 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.227442 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":262,"threads_started":1}
I20260811 03:05:16.229034 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.210191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:58930)
0811 03:05:16.211215 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.211263 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.211631 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.212221 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.212228 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.212970 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:16.213360 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.213380 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.214483 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.214489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.215844 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.215856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.216732 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.216740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.216977 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.217761 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:16.217800 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:16.220548 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:16.224169 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.224181 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.224196 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.224750 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.225196 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.225203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.225208 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.225513 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.226205 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:16.226214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:16.226679 (+ 465us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.227077 (+ 398us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.227762 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"spinlock_wait_cycles":24320,"thread_start_us":585,"threads_started":1}
I20260811 03:05:16.229382 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.207805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.27.109.60:42879)
0811 03:05:16.208770 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.208783 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.208826 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:05:16.211261 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.211293 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.211743 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.212077 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.213807 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.215126 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.216398 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.216982 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.217145 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.222049 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.222059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.222074 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.222107 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.225448 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.226334 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.226342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.226345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.226512 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.227038 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.227042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.227045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.227473 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.227821 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.228709 (+ 888us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.228913 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"spinlock_wait_cycles":19200,"thread_start_us":436,"threads_started":1}
I20260811 03:05:16.230832 3518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:05:16 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417516, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260811 03:05:16.246048 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.231807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.27.109.61:32823)
0811 03:05:16.232034 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.232041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.232063 (+ 22us) server_negotiation.cc:407] Connection header received
0811 03:05:16.232133 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.232138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.232226 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.232421 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.233256 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.234056 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.235525 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.236016 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.236166 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.239703 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.239708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.239712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.239749 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.242928 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.243729 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.243737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.243741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.243912 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.244355 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.244361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.244365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.244764 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.244943 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.245348 (+ 405us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.245623 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:05:16.246148 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.231499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.0.0.1:51576)
0811 03:05:16.231832 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.231851 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.231964 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.232444 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.232449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.232930 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:16.233085 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.233094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.234205 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.234214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.235349 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.235362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.236205 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.236211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.236420 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.237092 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:16.237111 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:16.239479 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:16.243097 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.243103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.243108 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.243591 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.244065 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.244070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.244074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.244230 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.244906 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:16.244912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:16.245086 (+ 174us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.245247 (+ 161us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.245623 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260811 03:05:16.247538 3518 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:05:16 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417516, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
I20260811 03:05:16.262154 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.248547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:42879 (local address 127.0.0.1:53330)
0811 03:05:16.248722 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.248740 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.248861 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.249189 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.249192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.249651 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:16.249817 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.249825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.251237 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.251241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.252160 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.252175 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.252958 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.252963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.253166 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.253854 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:16.253872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:16.256233 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:16.259609 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.259613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.259617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.259994 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.260491 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.260495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.260497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.260670 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.261316 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:16.261319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:16.261445 (+ 126us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.261598 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.261821 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 03:05:16.262344 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.248441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.27.109.60:42879)
0811 03:05:16.248644 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.248650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.248841 (+ 191us) server_negotiation.cc:407] Connection header received
0811 03:05:16.248981 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.248985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.249048 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.249139 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.250142 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.251101 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.252320 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.252764 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.252886 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.256456 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.256462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.256466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.256499 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.259446 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.260209 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.260214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.260216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.260361 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.260802 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.260806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.260809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.261197 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.261329 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.261711 (+ 382us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.262020 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":66}
I20260811 03:05:16.267807 3518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Bootstrap starting.
I20260811 03:05:16.271643 3518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:16.272868 3518 log.cc:824] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:16.276137 3518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: No bootstrap required, opened a new log
I20260811 03:05:16.283950 3518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } }
I20260811 03:05:16.284492 3518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:16.284643 3518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925210eabbaa41edb648ac0089514df7, State: Initialized, Role: FOLLOWER
I20260811 03:05:16.285321 3518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } }
I20260811 03:05:16.286325 3525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } } }
I20260811 03:05:16.286729 3525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:05:16.287531 3518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 03:05:16.299084 3537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:05:16.299273 3537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:05:16.305236 3353 master_runner.cc:428] Detected that this master 925210eabbaa41edb648ac0089514df7 is joining an existing cluster
I20260811 03:05:16.305408 3353 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.109.60:42879
I20260811 03:05:16.334832 3455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.310815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.27.109.61:32823)
0811 03:05:16.311065 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.311075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.315517 (+ 4442us) server_negotiation.cc:407] Connection header received
0811 03:05:16.316001 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.316011 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.316131 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.316228 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.319219 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.320406 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.322741 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.323408 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.323531 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.326538 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.326546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.326551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.326605 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.331057 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.332009 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.332016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.332020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.332198 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.332763 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.332768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.332772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.333180 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.333316 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.334109 (+ 793us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.334406 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 03:05:16.336202 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.310017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.0.0.1:51586)
0811 03:05:16.315318 (+ 5301us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.315352 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.315696 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.316569 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.316577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.317621 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:16.318123 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.318142 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.320626 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.320631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.322518 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.322531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.323869 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.323887 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.324075 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.324867 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:16.324910 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:16.326290 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:16.331236 (+ 4946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.331244 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.331249 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.331820 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.332365 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.332370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.332374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.332588 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.333431 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:16.333437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:16.333682 (+ 245us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.335193 (+ 1511us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.335577 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4530,"mutex_wait_us":31,"thread_start_us":16171,"threads_started":1}
I20260811 03:05:16.336512 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 2784
I20260811 03:05:16.336647 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.310588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.27.109.60:42879)
0811 03:05:16.311043 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.311049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.319096 (+ 8047us) server_negotiation.cc:407] Connection header received
0811 03:05:16.319370 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.319386 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.319452 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.319513 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.321122 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.322266 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.324280 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.325067 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.325164 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.328573 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.328581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.328588 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.328623 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.332659 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.333476 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.333481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.333484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.333670 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.334060 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.334064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.334067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.334475 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.334613 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.335579 (+ 966us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.335834 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":302}
I20260811 03:05:16.339711 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.310119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:42879 (local address 127.0.0.1:53336)
0811 03:05:16.318987 (+ 8868us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.319015 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.319249 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.319693 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.319698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.320444 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:16.320941 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.320961 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.322513 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.322517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.324114 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.324126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.325503 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.325509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.325673 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.326519 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:16.326542 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:16.328402 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:16.332836 (+ 4434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.332845 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.332849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.333347 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.333800 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.333805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.333808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.333950 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.334709 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:16.334713 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:16.334910 (+ 197us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.337113 (+ 2203us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.339032 (+ 1919us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8665,"mutex_wait_us":42,"thread_start_us":8373,"threads_started":1}
I20260811 03:05:16.347134 3525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:16.347487 3525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } }
I20260811 03:05:16.349426 3525 leader_election.cc:288] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3a23560c8fe4b3091ae72834ae3a9cc (127.27.109.62:41275), 00d96f2bcd114aa6a24abaf4352cd7c7 (127.27.109.61:32823)
I20260811 03:05:16.350526 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "925210eabbaa41edb648ac0089514df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" is_pre_election: true
I20260811 03:05:16.350984 2727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "925210eabbaa41edb648ac0089514df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" is_pre_election: true
I20260811 03:05:16.352604 3464 leader_election.cc:302] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 925210eabbaa41edb648ac0089514df7; no voters: 00d96f2bcd114aa6a24abaf4352cd7c7, a3a23560c8fe4b3091ae72834ae3a9cc
I20260811 03:05:16.353540 3525 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 11 03:05:16 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.1@KRBTEST.COM
I20260811 03:05:16.354961 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.310364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:58936)
0811 03:05:16.327720 (+ 17356us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.327745 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.327906 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.328430 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.328438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.329255 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:16.329727 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.329744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.331535 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.331541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.333289 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.333304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.337779 (+ 4475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.337787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.338066 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.338994 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:16.339031 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:16.342295 (+ 3264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:16.346989 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.346996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.347001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.347637 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.348240 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.348245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.348249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.348526 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.349348 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:16.349355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:16.353698 (+ 4343us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.353948 (+ 250us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.354384 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17114,"thread_start_us":4335,"threads_started":1}
I20260811 03:05:16.354990 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.309834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.27.109.62:41275)
0811 03:05:16.310071 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.310078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.327856 (+ 17778us) server_negotiation.cc:407] Connection header received
0811 03:05:16.328028 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.328033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.328140 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.328241 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.330079 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.331345 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.333483 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.334233 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.334340 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.342600 (+ 8260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.342611 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.342619 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.342702 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.346776 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.347853 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.347862 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.347868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.348075 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.348676 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.348682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.348685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.349093 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.349244 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.354151 (+ 4907us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.354539 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 03:05:16.365638 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.337783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.27.109.1:39493)
0811 03:05:16.340176 (+ 2393us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.340188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.340224 (+ 36us) server_negotiation.cc:407] Connection header received
0811 03:05:16.340318 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.340329 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.340672 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.340901 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.342796 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.344724 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.347841 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.348871 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.349774 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.355957 (+ 6183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.355993 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.356014 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.356064 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.360626 (+ 4562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.361730 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.361741 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.361745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.361924 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.362399 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.362404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.362408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.362891 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.363206 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.364863 (+ 1657us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.365180 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2168,"thread_start_us":1955,"threads_started":1}
I20260811 03:05:16.368515 2918 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 11 03:05:16 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
I20260811 03:05:16.376523 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.360812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.27.109.60:42879)
0811 03:05:16.361266 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.361271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.361284 (+ 13us) server_negotiation.cc:407] Connection header received
0811 03:05:16.361358 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.361362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.361463 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.361582 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.362357 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.363527 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.364813 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.365493 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.365642 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.369758 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.369769 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.369777 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.369828 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.373374 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.374369 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.374373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.374375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.374510 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.375017 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.375022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.375027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.375345 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.375518 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.375925 (+ 407us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.376150 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260811 03:05:16.376513 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.360677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:42879 (local address 127.0.0.1:53352)
0811 03:05:16.360969 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.360991 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.361099 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.361609 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.361616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.362009 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:16.362160 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.362196 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.363645 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.363651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.364671 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.364684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.365685 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.365692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.365894 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.366573 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:16.366601 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:16.369464 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:16.373542 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.373550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.373557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.374204 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.374648 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.374656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.374660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.374874 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.375481 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:16.375486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:16.375652 (+ 166us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.375804 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.376138 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260811 03:05:16.378530 2715 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.109.60:42879: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "925210eabbaa41edb648ac0089514df7"
member_type: NON_VOTER
last_known_addr {
host: "127.27.109.60"
port: 42879
}
attrs {
promote: true
}
}
dest_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc"
cas_config_opid_index: 5
I20260811 03:05:16.379510 2715 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } }
I20260811 03:05:16.381491 3453 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 925210eabbaa41edb648ac0089514df7 (127.27.109.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.382356 3489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:16.383519 3403 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 LEARNER]: Refusing update from remote peer a3a23560c8fe4b3091ae72834ae3a9cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 03:05:16.385921 3453 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.109.60:42879
I20260811 03:05:16.389652 3353 master.cc:537] Master@127.27.109.60:42879 shutting down...
I20260811 03:05:16.398965 3549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260811 03:05:16.400089 3453 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.400417 3453 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:16.411459 3403 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 925210eabbaa41edb648ac0089514df7 (127.27.109.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.416538 3431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: NON_VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: true } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.416873 3431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: This master's current role is: LEARNER
I20260811 03:05:16.417491 3550 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: attempting to promote NON_VOTER 00d96f2bcd114aa6a24abaf4352cd7c7 to VOTER
I20260811 03:05:16.418419 3550 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } }
I20260811 03:05:16.421433 3403 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 LEARNER]: Refusing update from remote peer a3a23560c8fe4b3091ae72834ae3a9cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260811 03:05:16.422107 3550 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 03:05:16.424721 3549 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 00d96f2bcd114aa6a24abaf4352cd7c7 (127.27.109.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.425575 3403 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 00d96f2bcd114aa6a24abaf4352cd7c7 (127.27.109.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.429448 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.429514 3453 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.429795 3453 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:16.429978 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:16.430913 3431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.431177 3431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 03:05:16.435151 3467 tablet.cc:2369] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:16.438367 3489 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 FOLLOWER]: Refusing update from remote peer a3a23560c8fe4b3091ae72834ae3a9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 03:05:16.442005 3353 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:16.442504 3353 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:16.442775 3353 tablet_replica.cc:330] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: stopping tablet replica
W20260811 03:05:16.442754 2700 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc -> Peer 925210eabbaa41edb648ac0089514df7 (127.27.109.60:42879): Couldn't send request to peer 925210eabbaa41edb648ac0089514df7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 03:05:16.451071 3353 master.cc:552] Master@127.27.109.60:42879 shutdown complete.
I20260811 03:05:16.451159 3353 master_runner.cc:305] Clearing existing system tablet
I20260811 03:05:16.451591 3353 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:16.451726 3353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:16.454136 3353 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260811 03:05:16.455775 3555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:16.456132 3353 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:16.456434 3353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 03:05:16.456559 3353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
uuid: "925210eabbaa41edb648ac0089514df7"
format_stamp: "Formatted at 2026-08-11 03:05:16 on dist-test-slave-8tdl"
server_key: "36abe30533b9b98d38c790cff6e7a229"
server_key_iv: "680efbd916dd1a4aa23fff2e82e4f9ee"
server_key_version: "encryptionkey@0"
I20260811 03:05:16.460914 3353 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:16.466152 3353 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 03:05:16.466358 3353 log.cc:1196] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260811 03:05:16.467022 3353 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Deleting consensus metadata
I20260811 03:05:16.467295 3353 master_runner.cc:315] Copying system tablet from 127.27.109.62:41275
I20260811 03:05:16.470021 3353 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: tablet copy: Beginning tablet copy session from remote peer at address 127.27.109.62:41275
I20260811 03:05:16.484212 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.470987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.27.109.62:41275)
0811 03:05:16.471275 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.471284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.471884 (+ 600us) server_negotiation.cc:407] Connection header received
0811 03:05:16.472136 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.472142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.472221 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.472334 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:16.473187 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.473952 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.475356 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.475824 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.475976 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.478060 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:16.478065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:16.478068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:16.478105 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:16.481194 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.481926 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.481932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.481936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.482094 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:16.482525 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:16.482531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:16.482534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:16.482955 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:16.483128 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.483572 (+ 444us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.483804 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260811 03:05:16.484660 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.470978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:58946)
0811 03:05:16.471757 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.471774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.471967 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.472371 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.472375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.472822 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:16.473030 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.473041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.474117 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.474123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.475201 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.475214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.475992 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.475998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.476182 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.476916 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:16.476935 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:16.477821 (+ 886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:16.481368 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.481374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.481378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.481777 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.482219 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:16.482224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:16.482228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:16.482402 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:16.483092 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:16.483096 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:16.483291 (+ 195us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.483446 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.483809 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":496,"threads_started":1}
I20260811 03:05:16.485517 2738 tablet_copy_service.cc:140] P a3a23560c8fe4b3091ae72834ae3a9cc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 925210eabbaa41edb648ac0089514df7 ({username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:58946)
I20260811 03:05:16.485723 2738 tablet_copy_service.cc:161] P a3a23560c8fe4b3091ae72834ae3a9cc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 925210eabbaa41edb648ac0089514df7 at {username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:58946: session id = 925210eabbaa41edb648ac0089514df7-00000000000000000000000000000000
I20260811 03:05:16.488382 2738 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 03:05:16.490307 3353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 03:05:16.496106 3353 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: tablet copy: Starting download of 0 data blocks...
I20260811 03:05:16.496500 3353 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: tablet copy: Starting download of 1 WAL segments...
I20260811 03:05:16.499099 3353 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 03:05:16.502840 2738 tablet_copy_service.cc:342] P a3a23560c8fe4b3091ae72834ae3a9cc: Request end of tablet copy session 925210eabbaa41edb648ac0089514df7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.109.60@KRBTEST.COM'} at 127.0.0.1:58946
I20260811 03:05:16.503016 2738 tablet_copy_service.cc:433] P a3a23560c8fe4b3091ae72834ae3a9cc: ending tablet copy session 925210eabbaa41edb648ac0089514df7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 925210eabbaa41edb648ac0089514df7
I20260811 03:05:16.512424 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 2784
I20260811 03:05:16.516285 3353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:16.519466 3564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:16.527765 3566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:16.535595 3353 server_base.cc:1034] running on GCE node
W20260811 03:05:16.536062 3568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:16.536582 3353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:16.537388 3353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:16.538625 3353 hybrid_clock.cc:648] HybridClock initialized: now 1786417516538611 us; error 69 us; skew 500 ppm
I20260811 03:05:16.538962 3353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:16 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417516, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:16.541227 3353 init.cc:377] Logged in from keytab as oryx/127.27.109.60@KRBTEST.COM (short username oryx)
I20260811 03:05:16.541817 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:39493
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:41275
--webserver_port=35907
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823,127.27.109.60:42879 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:05:16.548972 3353 webserver.cc:460] Webserver started at http://127.27.109.60:42411/ using document root <none> and password file <none>
I20260811 03:05:16.549402 3353 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:16.549549 3353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:16.552306 3353 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 03:05:16.554858 3579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:16.555418 3353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 03:05:16.555567 3353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
uuid: "925210eabbaa41edb648ac0089514df7"
format_stamp: "Formatted at 2026-08-11 03:05:16 on dist-test-slave-8tdl"
server_key: "36abe30533b9b98d38c790cff6e7a229"
server_key_iv: "680efbd916dd1a4aa23fff2e82e4f9ee"
server_key_version: "encryptionkey@0"
I20260811 03:05:16.555788 3353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:16.589619 3353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:16.594446 3353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:16.621376 3353 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:42879
I20260811 03:05:16.621434 3634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:42879 every 8 connection(s)
I20260811 03:05:16.622597 3353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
I20260811 03:05:16.627890 3635 sys_catalog.cc:263] Verifying existing consensus state
I20260811 03:05:16.631807 3635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Bootstrap starting.
I20260811 03:05:16.652701 3635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 03:05:16.653333 3635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: Bootstrap complete.
I20260811 03:05:16.654592 3635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } }
I20260811 03:05:16.654970 3635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 925210eabbaa41edb648ac0089514df7, State: Initialized, Role: LEARNER
I20260811 03:05:16.655334 3635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } }
I20260811 03:05:16.656495 3638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.656929 3635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:16.657013 3638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: This master's current role is: LEARNER
I20260811 03:05:16.668663 3651 catalog_manager.cc:1259] Loaded cluster ID: 64fdd41ffe424b3887d0b354c4dab236
I20260811 03:05:16.668829 3651 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: loading cluster ID for follower catalog manager: success
I20260811 03:05:16.674556 3651 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: acquiring CA information for follower catalog manager: success
I20260811 03:05:16.677640 3651 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 03:05:16.846338 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.815983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:42879 (local address 127.0.0.1:53366)
0811 03:05:16.816457 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:16.816483 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:16.816680 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:16.837024 (+ 20344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:16.837030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:16.837084 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:16.837300 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.837314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.840336 (+ 3022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.840346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.842817 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:16.842847 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.845468 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.845475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:16.845693 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.845737 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 03:05:16.845821 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 03:05:16.845990 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260811 03:05:16.848001 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:16.817066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.27.109.60:42879)
0811 03:05:16.836359 (+ 19293us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:16.836379 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:16.836418 (+ 39us) server_negotiation.cc:407] Connection header received
0811 03:05:16.836525 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:16.836532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:16.836641 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:16.837513 (+ 872us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:16.837856 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.840073 (+ 2217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.843208 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:16.845211 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:16.847120 (+ 1909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:16.847405 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:16.847474 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 03:05:16.847622 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19079,"thread_start_us":19711,"threads_started":1}
I20260811 03:05:16.849211 3608 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260811 03:05:16.850211 3638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3a23560c8fe4b3091ae72834ae3a9cc. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:16.850603 3638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: This master's current role is: LEARNER
W20260811 03:05:16.992668 3577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:16.993057 3577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:16.993248 3577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:17.001050 3577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:17.001235 3577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:17.001417 3577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:05:17.012173 3577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:39493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=35907
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823,127.27.109.60:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:17.014052 3577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:17.015657 3577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:17.034350 3659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:17.034807 3660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:17.035045 3662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:17.035983 3577 server_base.cc:1034] running on GCE node
I20260811 03:05:17.037003 3577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:17.038561 3577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:17.039976 3577 hybrid_clock.cc:648] HybridClock initialized: now 1786417517039890 us; error 69 us; skew 500 ppm
I20260811 03:05:17.040493 3577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417517, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:17.045684 3577 init.cc:377] Logged in from keytab as oryx/127.27.109.1@KRBTEST.COM (short username oryx)
I20260811 03:05:17.048173 3577 webserver.cc:460] Webserver started at http://127.27.109.1:35907/ using document root <none> and password file <none>
I20260811 03:05:17.049352 3577 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:17.049563 3577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:17.058490 3577 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:05:17.062736 3674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:17.064868 3577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:05:17.065109 3577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "6552402ca2054fbcb005b9ce1f80c2e5"
format_stamp: "Formatted at 2026-08-11 03:05:13 on dist-test-slave-8tdl"
server_key: "c6e9c37b68e94fb96885a42a223b73d6"
server_key_iv: "a609784202bdb471b4aeaaf4a4c47588"
server_key_version: "encryptionkey@0"
I20260811 03:05:17.067031 3577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:17.100350 3577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:17.105144 3577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:17.105474 3577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:17.106597 3577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:17.108299 3577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:17.108475 3577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:17.108633 3577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:17.108757 3577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:17.151506 3577 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:39493
I20260811 03:05:17.151566 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:39493 every 8 connection(s)
I20260811 03:05:17.153126 3577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:05:17.156639 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 3577
I20260811 03:05:17.157058 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 2932
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417517, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417517, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.2@KRBTEST.COM
I20260811 03:05:17.217804 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.157390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:49189 (local address 127.27.109.60:42879)
0811 03:05:17.157859 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:17.157866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:17.168464 (+ 10598us) server_negotiation.cc:407] Connection header received
0811 03:05:17.172770 (+ 4306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:17.172779 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:17.172911 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:17.173055 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:17.195076 (+ 22021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.196404 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.198884 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.199687 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.199841 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.208666 (+ 8825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:17.208676 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:17.208681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:17.208732 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:17.212525 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.213909 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.213915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.213919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.214114 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.214758 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.214764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.214769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.215202 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:17.215342 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:17.217041 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0811 03:05:17.217416 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260811 03:05:17.217903 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.158994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:49751 (local address 127.27.109.61:32823)
0811 03:05:17.187287 (+ 28293us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:17.187295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:17.188230 (+ 935us) server_negotiation.cc:407] Connection header received
0811 03:05:17.188308 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:17.188313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:17.188423 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:17.188525 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:17.190110 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.191398 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.195958 (+ 4560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.196876 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.197036 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.207761 (+ 10725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:17.207769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:17.207774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:17.207843 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:17.212544 (+ 4701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.214864 (+ 2320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.214872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.214876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.215032 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.215482 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.215487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.215490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.215845 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:17.216003 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:17.217238 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0811 03:05:17.217480 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28147,"thread_start_us":28299,"threads_started":1}
I20260811 03:05:17.219942 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.157361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:42879 (local address 127.27.109.1:49189)
0811 03:05:17.168182 (+ 10821us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:17.168278 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:17.172268 (+ 3990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:17.187146 (+ 14878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:17.187162 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:17.188215 (+ 1053us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:17.189255 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.189292 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.196655 (+ 7363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.196663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.198654 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.198699 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.200222 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.200229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.200558 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.201399 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:17.201423 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:17.208365 (+ 6942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:17.212745 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.212756 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.212779 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.213749 (+ 970us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.214288 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.214292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.214296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.214565 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.215436 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:17.215446 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:17.215955 (+ 509us) client_negotiation.cc:769] Sending connection context
0811 03:05:17.218722 (+ 2767us) client_negotiation.cc:240] Negotiation successful
0811 03:05:17.219051 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10322,"thread_start_us":10039,"threads_started":1}
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417517, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:17.220003 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.159380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.27.109.1:49751)
0811 03:05:17.187895 (+ 28515us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:17.187921 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:17.188095 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:17.188778 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:17.188782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:17.189484 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:17.189856 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.189871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.192546 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.192555 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.195707 (+ 3152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.195730 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.197514 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.197523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.197944 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.200222 (+ 2278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:17.200255 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:17.207490 (+ 7235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:17.212745 (+ 5255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.212756 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.212779 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.214695 (+ 1916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.215183 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.215187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.215190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.215351 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.216069 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:17.216074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:17.216209 (+ 135us) client_negotiation.cc:769] Sending connection context
0811 03:05:17.216395 (+ 186us) client_negotiation.cc:240] Negotiation successful
0811 03:05:17.218345 (+ 1950us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28372,"spinlock_wait_cycles":28416,"thread_start_us":35299,"threads_started":1}
I20260811 03:05:17.223771 3795 heartbeater.cc:347] Connected to a master server at 127.27.109.60:42879
I20260811 03:05:17.224248 3795 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:17.225232 3795 heartbeater.cc:511] Master 127.27.109.60:42879 requested a full tablet report, sending...
I20260811 03:05:17.228762 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.159010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.27.109.2:45191)
0811 03:05:17.200166 (+ 41156us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:17.200179 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:17.200213 (+ 34us) server_negotiation.cc:407] Connection header received
0811 03:05:17.200291 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:17.200303 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:17.200634 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:17.202832 (+ 2198us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:17.203979 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.205914 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.208445 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.209590 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.210497 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.217296 (+ 6799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:17.217318 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:17.217341 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:17.217418 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:17.222195 (+ 4777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.223313 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.223323 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.223326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.223565 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.224160 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.224166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.224169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.225657 (+ 1488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:17.226023 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:17.228037 (+ 2014us) server_negotiation.cc:299] Negotiation successful
0811 03:05:17.228335 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40959,"thread_start_us":40798,"threads_started":1}
I20260811 03:05:17.229372 3797 heartbeater.cc:347] Connected to a master server at 127.27.109.61:32823
I20260811 03:05:17.229516 3797 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:17.230082 3797 heartbeater.cc:511] Master 127.27.109.61:32823 requested a full tablet report, sending...
I20260811 03:05:17.230995 3070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:17.232402 3393 ts_manager.cc:194] Registered new tserver with Master: 6552402ca2054fbcb005b9ce1f80c2e5 (127.27.109.1:39493)
I20260811 03:05:17.234170 3597 ts_manager.cc:194] Registered new tserver with Master: 6552402ca2054fbcb005b9ce1f80c2e5 (127.27.109.1:39493)
I20260811 03:05:17.239471 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.157946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57775 (local address 127.27.109.62:41275)
0811 03:05:17.167247 (+ 9301us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:17.167259 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:17.199507 (+ 32248us) server_negotiation.cc:407] Connection header received
0811 03:05:17.199683 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:17.199692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:17.199800 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:17.199935 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:17.204172 (+ 4237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.205181 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.213401 (+ 8220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.214148 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.214282 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.224573 (+ 10291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:17.224584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:17.224590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:17.224639 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:17.229072 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.230244 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.230250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.230253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.230427 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.237284 (+ 6857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.237295 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.237301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.237827 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:17.238063 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:17.238633 (+ 570us) server_negotiation.cc:299] Negotiation successful
0811 03:05:17.238981 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9095,"thread_start_us":8927,"threads_started":1}
I20260811 03:05:17.239567 3806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.160969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.1:57775)
0811 03:05:17.199372 (+ 38403us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:17.199408 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:17.199557 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:17.200894 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:17.200899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:17.203572 (+ 2673us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:17.203950 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.203963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.205331 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.205340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.213229 (+ 7889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.213245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.216410 (+ 3165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.216417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.216602 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.220269 (+ 3667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:17.220300 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:17.224271 (+ 3971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:17.229311 (+ 5040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.229320 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.229326 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.230062 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.236802 (+ 6740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.236811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.236819 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.237091 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.237996 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:17.238005 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:17.238254 (+ 249us) client_negotiation.cc:769] Sending connection context
0811 03:05:17.238442 (+ 188us) client_negotiation.cc:240] Negotiation successful
0811 03:05:17.238944 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38228,"thread_start_us":38018,"threads_started":1}
I20260811 03:05:17.240549 3796 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41275
I20260811 03:05:17.240716 3796 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:17.241182 3796 heartbeater.cc:511] Master 127.27.109.62:41275 requested a full tablet report, sending...
I20260811 03:05:17.242551 2715 ts_manager.cc:194] Re-registered known tserver with Master: 6552402ca2054fbcb005b9ce1f80c2e5 (127.27.109.1:39493)
I20260811 03:05:17.243604 2715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:57775
I20260811 03:05:17.332415 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 2932
I20260811 03:05:17.358119 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:45191
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:41275
--webserver_port=42887
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823,127.27.109.60:42879 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:05:17.422030 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: NON_VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: true } } }
I20260811 03:05:17.422600 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:17.424245 3811 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: attempting to promote NON_VOTER 925210eabbaa41edb648ac0089514df7 to VOTER
I20260811 03:05:17.424955 3811 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } }
I20260811 03:05:17.427500 3403 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Refusing update from remote peer a3a23560c8fe4b3091ae72834ae3a9cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260811 03:05:17.427793 3608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 LEARNER]: Refusing update from remote peer a3a23560c8fe4b3091ae72834ae3a9cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260811 03:05:17.428439 3812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260811 03:05:17.429215 3550 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260811 03:05:17.432989 3550 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 925210eabbaa41edb648ac0089514df7 (127.27.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:17.436872 3403 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 925210eabbaa41edb648ac0089514df7 (127.27.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:17.438181 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:17.438575 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:17.439252 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:17.439584 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:17.440227 3812 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:17.440474 3812 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:17.440474 3607 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 925210eabbaa41edb648ac0089514df7 (127.27.109.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:17.446278 3818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:17.446763 3818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:05:17.447366 3817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:17.448072 3817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 03:05:17.752452 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:17.752771 3810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:17.752868 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:17.761230 3810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:17.761365 3810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:17.761487 3810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:17.772115 3810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:45191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=42887
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823,127.27.109.60:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:17.773895 3810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:17.775801 3810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:17.790161 3823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:17.790231 3827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:17.790138 3824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:17.791389 3810 server_base.cc:1034] running on GCE node
I20260811 03:05:17.792138 3810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:17.793187 3810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:17.794456 3810 hybrid_clock.cc:648] HybridClock initialized: now 1786417517794388 us; error 58 us; skew 500 ppm
I20260811 03:05:17.794901 3810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417517, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:17.803673 3810 init.cc:377] Logged in from keytab as oryx/127.27.109.2@KRBTEST.COM (short username oryx)
I20260811 03:05:17.805593 3810 webserver.cc:460] Webserver started at http://127.27.109.2:42887/ using document root <none> and password file <none>
I20260811 03:05:17.806396 3810 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:17.806567 3810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:17.813361 3810 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260811 03:05:17.816919 3836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:17.818563 3810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:05:17.818790 3810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "863bd409cd0d4058bdc16227e559e379"
format_stamp: "Formatted at 2026-08-11 03:05:14 on dist-test-slave-8tdl"
server_key: "b869650bc8d012645ce2ccf87cb00cce"
server_key_iv: "c13b5f614c31c551a7ab2d1178abdb38"
server_key_version: "encryptionkey@0"
I20260811 03:05:17.819940 3810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:17.847846 3810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:17.852985 3810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:17.853338 3810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:17.854480 3810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:17.856179 3810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:17.856302 3810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:17.856436 3810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:17.856526 3810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:17.895988 3810 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45191
I20260811 03:05:17.896111 3959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45191 every 8 connection(s)
I20260811 03:05:17.897603 3810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:17.900522 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 3810
I20260811 03:05:17.900827 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 3081
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417517, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417517, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417517, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
Aug 11 03:05:17 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.3@KRBTEST.COM
I20260811 03:05:17.943745 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.901217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:49359 (local address 127.27.109.60:42879)
0811 03:05:17.916339 (+ 15122us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:17.916348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:17.916366 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:05:17.916432 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:17.916441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:17.916503 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:17.916634 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:17.921263 (+ 4629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.922737 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.928298 (+ 5561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.929016 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.929171 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.935018 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:17.935027 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:17.935034 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:17.935090 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:17.938702 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.940079 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.940087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.940093 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.940269 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.941172 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.941177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.941181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.941607 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:17.941744 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:17.943011 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0811 03:05:17.943289 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14971,"thread_start_us":14854,"threads_started":1}
I20260811 03:05:17.945102 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.902532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:38149 (local address 127.27.109.62:41275)
0811 03:05:17.915184 (+ 12652us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:17.915189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:17.915766 (+ 577us) server_negotiation.cc:407] Connection header received
0811 03:05:17.915981 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:17.915989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:17.916102 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:17.916218 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:17.920140 (+ 3922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.921368 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.926474 (+ 5106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.927232 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.927366 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.935025 (+ 7659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:17.935033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:17.935038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:17.935099 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:17.941215 (+ 6116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.942155 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.942163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.942166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.942350 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.942907 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.942912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.942915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.943438 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:17.943595 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:17.944394 (+ 799us) server_negotiation.cc:299] Negotiation successful
0811 03:05:17.944677 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12527,"thread_start_us":12381,"threads_started":1}
I20260811 03:05:17.946389 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.902084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:42879 (local address 127.27.109.2:49359)
0811 03:05:17.911606 (+ 9522us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:17.911702 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:17.913766 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:17.920170 (+ 6404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:17.920174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:17.920741 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:17.921047 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.921058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.926530 (+ 5472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.926534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.928077 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.928096 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.929241 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.929248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.929429 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.930328 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:17.930355 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:17.934641 (+ 4286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:17.938903 (+ 4262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.938916 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.938940 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.939900 (+ 960us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.940442 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.940449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.940453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.941020 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.941866 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:17.941875 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:17.942413 (+ 538us) client_negotiation.cc:769] Sending connection context
0811 03:05:17.944047 (+ 1634us) client_negotiation.cc:240] Negotiation successful
0811 03:05:17.944701 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9158,"thread_start_us":8923,"threads_started":1}
I20260811 03:05:17.946518 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.902085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.2:38149)
0811 03:05:17.915637 (+ 13552us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:17.915658 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:17.915840 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:17.917222 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:17.917243 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:17.918600 (+ 1357us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:17.919886 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.919924 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.923068 (+ 3144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.923077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.926260 (+ 3183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.926288 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.928166 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.928180 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.928496 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.929835 (+ 1339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:17.929880 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:17.934641 (+ 4761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:17.941398 (+ 6757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.941404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.941420 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.941989 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.942524 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.942531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.942538 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.942761 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.943666 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:17.943672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:17.943852 (+ 180us) client_negotiation.cc:769] Sending connection context
0811 03:05:17.944017 (+ 165us) client_negotiation.cc:240] Negotiation successful
0811 03:05:17.945675 (+ 1658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13403,"mutex_wait_us":936,"thread_start_us":12258,"threads_started":1}
I20260811 03:05:17.948477 3960 heartbeater.cc:347] Connected to a master server at 127.27.109.60:42879
I20260811 03:05:17.948861 3960 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:17.949286 3973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.916863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:52351 (local address 127.27.109.61:32823)
0811 03:05:17.927763 (+ 10900us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:17.927772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:17.928465 (+ 693us) server_negotiation.cc:407] Connection header received
0811 03:05:17.928671 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:17.928677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:17.928769 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:17.929264 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:17.931580 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.932444 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.934236 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.934997 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.935099 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.940334 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:17.940345 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:17.940352 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:17.940410 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:17.944454 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.945676 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.945682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.945685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.945848 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.946351 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.946356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.946358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.947563 (+ 1205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:17.948500 (+ 937us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:17.948679 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 03:05:17.948928 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10700,"thread_start_us":10651,"threads_started":1}
I20260811 03:05:17.949311 3974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.916972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.27.109.2:52351)
0811 03:05:17.928335 (+ 11363us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:17.928364 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:17.928520 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:17.930037 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:17.930043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:17.930935 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:17.931383 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.931404 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.932604 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.932611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.934049 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:17.934066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.936052 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.936060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:17.936305 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.936892 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:17.936928 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:17.939973 (+ 3045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:17.944620 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.944625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.944629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.945503 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.946008 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:17.946013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:17.946016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:17.946206 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:17.947721 (+ 1515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:17.947726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:17.947890 (+ 164us) client_negotiation.cc:769] Sending connection context
0811 03:05:17.948554 (+ 664us) client_negotiation.cc:240] Negotiation successful
0811 03:05:17.948877 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11203,"thread_start_us":11640,"threads_started":1}
I20260811 03:05:17.949889 3960 heartbeater.cc:511] Master 127.27.109.60:42879 requested a full tablet report, sending...
I20260811 03:05:17.950238 3961 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41275
I20260811 03:05:17.950405 3961 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:17.951072 3961 heartbeater.cc:511] Master 127.27.109.62:41275 requested a full tablet report, sending...
I20260811 03:05:17.951699 3962 heartbeater.cc:347] Connected to a master server at 127.27.109.61:32823
I20260811 03:05:17.951829 3962 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:17.952119 3597 ts_manager.cc:194] Registered new tserver with Master: 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191)
I20260811 03:05:17.952231 3962 heartbeater.cc:511] Master 127.27.109.61:32823 requested a full tablet report, sending...
I20260811 03:05:17.952333 2715 ts_manager.cc:194] Re-registered known tserver with Master: 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191)
I20260811 03:05:17.953474 2715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:38149
I20260811 03:05:17.953917 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:17.902619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.27.109.3:40047)
0811 03:05:17.926817 (+ 24198us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:17.926832 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:17.927869 (+ 1037us) server_negotiation.cc:407] Connection header received
0811 03:05:17.927970 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:17.927983 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:17.928323 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:17.929682 (+ 1359us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:17.931709 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.933430 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.935856 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:17.936914 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:17.937067 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:17.943764 (+ 6697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:17.943785 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:17.944090 (+ 305us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:17.944153 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:17.948633 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.950293 (+ 1660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.950305 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.950308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.950496 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:17.951107 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:17.951111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:17.951113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:17.952569 (+ 1456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:17.952902 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:17.953270 (+ 368us) server_negotiation.cc:299] Negotiation successful
0811 03:05:17.953499 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23922,"thread_start_us":23668,"threads_started":1}
I20260811 03:05:17.954805 3393 ts_manager.cc:194] Registered new tserver with Master: 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191)
I20260811 03:05:17.955338 3222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:18.043727 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 3081
I20260811 03:05:18.065912 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:40047
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:41275
--webserver_port=34557
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823,127.27.109.60:42879 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:05:18.247117 3796 heartbeater.cc:503] Master 127.27.109.62:41275 was elected leader, sending a full tablet report...
W20260811 03:05:18.447335 3980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:18.447733 3980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:18.448038 3980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:18.458669 3980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:18.458859 3980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:18.459059 3980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:18.471113 3980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:40047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=34557
--tserver_master_addrs=127.27.109.62:41275,127.27.109.61:32823,127.27.109.60:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:18.472864 3980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:18.474530 3980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:18.490269 3988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:18.490239 3985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:18.490326 3986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:18.491192 3980 server_base.cc:1034] running on GCE node
I20260811 03:05:18.491868 3980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:18.492880 3980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:18.494125 3980 hybrid_clock.cc:648] HybridClock initialized: now 1786417518494071 us; error 48 us; skew 500 ppm
I20260811 03:05:18.494467 3980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:18 dist-test-slave-8tdl krb5kdc[2665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417518, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:18.497990 3980 init.cc:377] Logged in from keytab as oryx/127.27.109.3@KRBTEST.COM (short username oryx)
I20260811 03:05:18.499737 3980 webserver.cc:460] Webserver started at http://127.27.109.3:34557/ using document root <none> and password file <none>
I20260811 03:05:18.500488 3980 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:18.500689 3980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:18.506657 3980 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:05:18.509923 3997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:18.511591 3980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 03:05:18.511811 3980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "39af5b9dd1f44f419bb5374224bff222"
format_stamp: "Formatted at 2026-08-11 03:05:14 on dist-test-slave-8tdl"
server_key: "1eee159709aff5390b135889e0ce4443"
server_key_iv: "6b19bc5c77e65b220ff3892fa121dc13"
server_key_version: "encryptionkey@0"
I20260811 03:05:18.512945 3980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:18.529924 3980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:18.534986 3980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:18.535317 3980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:18.536391 3980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:18.537880 3980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:18.538008 3980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:18.538148 3980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:18.538244 3980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:18.576013 3980 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:40047
I20260811 03:05:18.576112 4124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:40047 every 8 connection(s)
I20260811 03:05:18.577581 3980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:18.578756 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 3980
Aug 11 03:05:18 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417518, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.60@KRBTEST.COM
Aug 11 03:05:18 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417518, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:18.626895 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.581056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:33365 (local address 127.27.109.60:42879)
0811 03:05:18.593524 (+ 12468us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.593534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.599582 (+ 6048us) server_negotiation.cc:407] Connection header received
0811 03:05:18.599776 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.599782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.599878 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.599980 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:18.602259 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.603505 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.606367 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.607073 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.607202 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.612569 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:18.612579 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:18.612587 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:18.612642 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:18.617077 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.618535 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.618544 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.618551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.618816 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.619543 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.619551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.619558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.620045 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:18.620217 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.623603 (+ 3386us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.626409 (+ 2806us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12255,"thread_start_us":12479,"threads_started":1}
I20260811 03:05:18.630404 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.582090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:42879 (local address 127.27.109.3:33365)
0811 03:05:18.599424 (+ 17334us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:18.599457 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:18.599630 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:18.600452 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:18.600461 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:18.601313 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:18.602016 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.602035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.603728 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.603734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.606177 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.606191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.607554 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.607564 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.607859 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.608740 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:18.608780 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:18.612250 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:18.617305 (+ 5055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:18.617323 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:18.617351 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:18.618323 (+ 972us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:18.619003 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:18.619011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:18.619018 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:18.619370 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:18.620322 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:18.621304 (+ 982us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:18.621913 (+ 609us) client_negotiation.cc:769] Sending connection context
0811 03:05:18.622406 (+ 493us) client_negotiation.cc:240] Negotiation successful
0811 03:05:18.623173 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17160,"thread_start_us":11799,"threads_started":1}
I20260811 03:05:18.634246 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.581060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:46611 (local address 127.27.109.62:41275)
0811 03:05:18.602991 (+ 21931us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.602998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.603019 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:05:18.603079 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.603085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.603160 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.603269 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:18.604550 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.605686 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.607598 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.608707 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.608843 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.624761 (+ 15918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:18.624773 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:18.624781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:18.624906 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:18.631120 (+ 6214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.631895 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.631906 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.631913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.632124 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.632550 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.632558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.632564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.633057 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:18.633227 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.633642 (+ 415us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.633880 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21779,"thread_start_us":21728,"threads_started":1}
I20260811 03:05:18.634286 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.581483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.3:46611)
0811 03:05:18.594391 (+ 12908us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:18.594450 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:18.598228 (+ 3778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:18.603454 (+ 5226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:18.603459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:18.604013 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:18.604350 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.604364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.605884 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.605889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.607410 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.607423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.609182 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.609191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.609409 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.610064 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:18.610086 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:18.624364 (+ 14278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:18.631350 (+ 6986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:18.631355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:18.631358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:18.631735 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:18.632234 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:18.632239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:18.632243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:18.632407 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:18.633178 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:18.633182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:18.633352 (+ 170us) client_negotiation.cc:769] Sending connection context
0811 03:05:18.633492 (+ 140us) client_negotiation.cc:240] Negotiation successful
0811 03:05:18.633836 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12543,"thread_start_us":17417,"threads_started":1}
I20260811 03:05:18.636909 4125 heartbeater.cc:347] Connected to a master server at 127.27.109.60:42879
I20260811 03:05:18.637269 4125 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:18.638300 4125 heartbeater.cc:511] Master 127.27.109.60:42879 requested a full tablet report, sending...
I20260811 03:05:18.639638 4127 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41275
I20260811 03:05:18.639791 4127 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:18.640264 4127 heartbeater.cc:511] Master 127.27.109.62:41275 requested a full tablet report, sending...
I20260811 03:05:18.641568 3597 ts_manager.cc:194] Registered new tserver with Master: 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047)
I20260811 03:05:18.643518 2715 ts_manager.cc:194] Re-registered known tserver with Master: 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047)
I20260811 03:05:18.644709 2715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:46611
Aug 11 03:05:18 dist-test-slave-8tdl krb5kdc[2665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417518, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.61@KRBTEST.COM
I20260811 03:05:18.658089 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.612259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.27.109.62:41275)
0811 03:05:18.635017 (+ 22758us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.635022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.635040 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:05:18.635087 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.635091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.635176 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.639586 (+ 4410us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:18.639746 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.640874 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.642765 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.643517 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.643643 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.648896 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:18.648905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:18.648913 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:18.648972 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:18.652671 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.653913 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.653920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.653923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.654101 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.655513 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.655520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.655523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.656009 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:18.656291 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.656622 (+ 331us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.656886 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22632,"thread_start_us":26722,"threads_started":1}
I20260811 03:05:18.659111 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.613995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.27.109.3:45541)
0811 03:05:18.627153 (+ 13158us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:18.627177 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:18.627294 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:18.638317 (+ 11023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:18.638322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:18.638957 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:18.639302 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.639320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.643109 (+ 3789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.643117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.644478 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.644496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.646335 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.646343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.646528 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.647115 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:18.647139 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:18.650360 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:18.654670 (+ 4310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:18.654707 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:18.654711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:18.655186 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:18.656512 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:18.656517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:18.656521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:18.656774 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:18.657918 (+ 1144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:18.657924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:18.658116 (+ 192us) client_negotiation.cc:769] Sending connection context
0811 03:05:18.658277 (+ 161us) client_negotiation.cc:240] Negotiation successful
0811 03:05:18.658734 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13015,"thread_start_us":12692,"threads_started":1}
I20260811 03:05:18.659255 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.613967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:45541 (local address 127.27.109.61:32823)
0811 03:05:18.637108 (+ 23141us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.637114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.637135 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:05:18.637198 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.637205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.637296 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.637424 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:18.639556 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.641100 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.645353 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.646104 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.646865 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.650627 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:18.650637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:18.650648 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:18.650737 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:18.654525 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.656112 (+ 1587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.656117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.656121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.656293 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.657394 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.657399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.657404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.657770 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:18.657929 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.658491 (+ 562us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.658792 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23007,"thread_start_us":25078,"threads_started":1}
I20260811 03:05:18.661576 4128 heartbeater.cc:347] Connected to a master server at 127.27.109.61:32823
I20260811 03:05:18.661736 4128 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:18.662209 4128 heartbeater.cc:511] Master 127.27.109.61:32823 requested a full tablet report, sending...
I20260811 03:05:18.663611 3393 ts_manager.cc:194] Registered new tserver with Master: 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047)
I20260811 03:05:18.665223 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.639656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.27.109.60:42879)
0811 03:05:18.639858 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.639865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.645742 (+ 5877us) server_negotiation.cc:407] Connection header received
0811 03:05:18.646024 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.646032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.646159 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.646240 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:18.648612 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.649817 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.651817 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.652474 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.652587 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.656286 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:18.656293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:18.656298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:18.656340 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:18.660443 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.662668 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.662697 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.662700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.662868 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.663344 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.663349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.663352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.663737 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:18.663972 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.664584 (+ 612us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.664845 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 03:05:18.668311 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.620759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.27.109.61:32823)
0811 03:05:18.644227 (+ 23468us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.644233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.644253 (+ 20us) server_negotiation.cc:407] Connection header received
0811 03:05:18.644303 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.644308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.644382 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.644489 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:18.650487 (+ 5998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.652214 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.655138 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.655773 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.655895 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.661682 (+ 5787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:18.661688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:18.661693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:18.661731 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:18.665480 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.666129 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.666134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.666138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.666290 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.666729 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.666734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.666738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.667032 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:18.667299 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.667631 (+ 332us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.667931 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23315,"thread_start_us":23873,"threads_started":1}
I20260811 03:05:18.663213 2715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:05:18.672430 2715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:05:18.720705 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.700738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.27.109.3:40047)
0811 03:05:18.713495 (+ 12757us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.713502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.713532 (+ 30us) server_negotiation.cc:407] Connection header received
0811 03:05:18.713620 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.713626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.713827 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.714065 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:18.714815 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.715904 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.718007 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.719319 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.719463 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.719703 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.720148 (+ 445us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.720334 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12576,"thread_start_us":12388,"threads_started":1}
I20260811 03:05:18.721375 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.700425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:40047 (local address 127.0.0.1:39222)
0811 03:05:18.712397 (+ 11972us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:18.712427 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:18.712599 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:18.714055 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:18.714060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:18.714093 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:18.714567 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.714577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.716067 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.716075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.717819 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.717855 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.719602 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.719611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.719888 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.719947 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 03:05:18.720072 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 03:05:18.720283 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11768,"mutex_wait_us":2,"threads_started":1}
I20260811 03:05:18.723412 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.699380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.27.109.1:39493)
0811 03:05:18.699969 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.699981 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.701132 (+ 1151us) server_negotiation.cc:407] Connection header received
0811 03:05:18.701288 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.701301 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.708034 (+ 6733us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.708352 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:18.710477 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.712625 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.718945 (+ 6320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.720361 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.720516 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.720735 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.722864 (+ 2129us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.723081 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":286,"threads_started":1}
I20260811 03:05:18.723161 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.699101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39493 (local address 127.0.0.1:50408)
0811 03:05:18.700758 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:18.700791 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:18.700966 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:18.708710 (+ 7744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:18.708720 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:18.708763 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:18.709266 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.709283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.712872 (+ 3589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.712881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.718617 (+ 5736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.718640 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.722056 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.722066 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.722349 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.722408 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 03:05:18.722560 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 03:05:18.722766 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":1245,"thread_start_us":974,"threads_started":1}
I20260811 03:05:18.727818 3727 tablet_service.cc:1467] Processing CreateTablet for tablet 58c66a22ff4f420a849d935eedecebd1 (DEFAULT_TABLE table=test-table [id=ade05b6177f0460f81223b0b31b65678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:18.727818 4053 tablet_service.cc:1467] Processing CreateTablet for tablet 58c66a22ff4f420a849d935eedecebd1 (DEFAULT_TABLE table=test-table [id=ade05b6177f0460f81223b0b31b65678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:18.729137 4053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c66a22ff4f420a849d935eedecebd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:18.729183 3727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c66a22ff4f420a849d935eedecebd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:18.736580 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.701941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.27.109.2:45191)
0811 03:05:18.722292 (+ 20351us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.722300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.723359 (+ 1059us) server_negotiation.cc:407] Connection header received
0811 03:05:18.723694 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.723704 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.724042 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.724324 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:18.729556 (+ 5232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.731252 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.733845 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.735391 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.735558 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.735789 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.736077 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.736269 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20182,"thread_start_us":20342,"threads_started":1}
I20260811 03:05:18.736579 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.701646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45191 (local address 127.0.0.1:60972)
0811 03:05:18.723218 (+ 21572us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:18.723241 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:18.723390 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:18.724597 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:18.724601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:18.724648 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:18.724995 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.725004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.731452 (+ 6448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.731458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.733632 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.733652 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.735639 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.735646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.735854 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.735895 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 03:05:18.736016 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 03:05:18.736176 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21123,"thread_start_us":11062,"threads_started":1}
I20260811 03:05:18.741051 3891 tablet_service.cc:1467] Processing CreateTablet for tablet 58c66a22ff4f420a849d935eedecebd1 (DEFAULT_TABLE table=test-table [id=ade05b6177f0460f81223b0b31b65678]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:18.742715 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58c66a22ff4f420a849d935eedecebd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:18.744364 4166 tablet_bootstrap.cc:492] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: Bootstrap starting.
I20260811 03:05:18.748271 4166 tablet_bootstrap.cc:654] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:18.749272 4167 tablet_bootstrap.cc:492] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Bootstrap starting.
I20260811 03:05:18.749786 4166 log.cc:824] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:18.753489 4167 tablet_bootstrap.cc:654] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:18.753595 4166 tablet_bootstrap.cc:492] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: No bootstrap required, opened a new log
I20260811 03:05:18.753895 4166 ts_tablet_manager.cc:1397] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260811 03:05:18.755136 4167 log.cc:824] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:18.759039 4167 tablet_bootstrap.cc:492] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: No bootstrap required, opened a new log
I20260811 03:05:18.759392 4167 ts_tablet_manager.cc:1397] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260811 03:05:18.760432 4172 tablet_bootstrap.cc:492] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: Bootstrap starting.
I20260811 03:05:18.764195 4172 tablet_bootstrap.cc:654] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:18.765326 4172 log.cc:824] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:18.765878 4166 raft_consensus.cc:348] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.766381 4166 raft_consensus.cc:374] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:18.766497 4166 raft_consensus.cc:729] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6552402ca2054fbcb005b9ce1f80c2e5, State: Initialized, Role: FOLLOWER
I20260811 03:05:18.767269 4166 consensus_queue.cc:260] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.768985 4172 tablet_bootstrap.cc:492] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: No bootstrap required, opened a new log
I20260811 03:05:18.769268 4166 ts_tablet_manager.cc:1428] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260811 03:05:18.769361 4172 ts_tablet_manager.cc:1397] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 03:05:18.773387 4167 raft_consensus.cc:348] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.773917 4167 raft_consensus.cc:374] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:18.774044 4167 raft_consensus.cc:729] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39af5b9dd1f44f419bb5374224bff222, State: Initialized, Role: FOLLOWER
I20260811 03:05:18.774750 4167 consensus_queue.cc:260] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.776788 4167 ts_tablet_manager.cc:1428] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260811 03:05:18.777117 4127 heartbeater.cc:503] Master 127.27.109.62:41275 was elected leader, sending a full tablet report...
I20260811 03:05:18.777892 4174 raft_consensus.cc:482] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:18.778189 4174 raft_consensus.cc:504] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.785768 4172 raft_consensus.cc:348] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.786293 4172 raft_consensus.cc:374] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:18.786448 4172 raft_consensus.cc:729] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863bd409cd0d4058bdc16227e559e379, State: Initialized, Role: FOLLOWER
I20260811 03:05:18.787602 4172 consensus_queue.cc:260] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.790485 4172 ts_tablet_manager.cc:1428] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: Time spent starting tablet: real 0.021s user 0.012s sys 0.000s
I20260811 03:05:18.790869 3961 heartbeater.cc:503] Master 127.27.109.62:41275 was elected leader, sending a full tablet report...
I20260811 03:05:18.794143 4174 leader_election.cc:288] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047), 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191)
I20260811 03:05:18.796574 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.787116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:54079 (local address 127.27.109.3:40047)
0811 03:05:18.787396 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.787406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.788734 (+ 1328us) server_negotiation.cc:407] Connection header received
0811 03:05:18.789057 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.789063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.789147 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.789216 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:18.790071 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.791150 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.793859 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.795218 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.795351 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.795441 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.796145 (+ 704us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.796300 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 03:05:18.797340 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.787043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:40047 (local address 127.27.109.1:54079)
0811 03:05:18.788603 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:18.788628 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:18.788867 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:18.789451 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:18.789456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:18.789498 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:18.789882 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.789895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.791358 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.791363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.793652 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.793670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.795679 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.795685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.795909 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.795953 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 03:05:18.796827 (+ 874us) client_negotiation.cc:240] Negotiation successful
0811 03:05:18.796981 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1392,"thread_start_us":1186,"threads_started":1}
I20260811 03:05:18.798518 4073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c66a22ff4f420a849d935eedecebd1" candidate_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39af5b9dd1f44f419bb5374224bff222" is_pre_election: true
I20260811 03:05:18.799157 4073 raft_consensus.cc:2454] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6552402ca2054fbcb005b9ce1f80c2e5 in term 0.
I20260811 03:05:18.800038 3678 leader_election.cc:302] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39af5b9dd1f44f419bb5374224bff222, 6552402ca2054fbcb005b9ce1f80c2e5; no voters:
I20260811 03:05:18.800516 4174 raft_consensus.cc:2781] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:18.800669 4174 raft_consensus.cc:482] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:18.800770 4174 raft_consensus.cc:3037] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:18.801793 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.788134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:58651 (local address 127.27.109.2:45191)
0811 03:05:18.788406 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.788415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.795008 (+ 6593us) server_negotiation.cc:407] Connection header received
0811 03:05:18.795295 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.795300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.795374 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.795431 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:18.796224 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.797225 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.799473 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.800751 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.800856 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.800906 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.801397 (+ 491us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.801537 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 03:05:18.801867 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.787745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45191 (local address 127.27.109.1:58651)
0811 03:05:18.794890 (+ 7145us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:18.794913 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:18.795132 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:18.795630 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:18.795636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:18.795670 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:18.796016 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.796030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.797403 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.797409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.799246 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:18.799271 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.801109 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.801115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:18.801235 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.801258 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 03:05:18.801367 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 03:05:18.801509 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6970,"thread_start_us":6777,"threads_started":1}
I20260811 03:05:18.803162 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c66a22ff4f420a849d935eedecebd1" candidate_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863bd409cd0d4058bdc16227e559e379" is_pre_election: true
I20260811 03:05:18.803732 3912 raft_consensus.cc:2454] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6552402ca2054fbcb005b9ce1f80c2e5 in term 0.
I20260811 03:05:18.804201 4174 raft_consensus.cc:504] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.805413 4174 leader_election.cc:288] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [CANDIDATE]: Term 1 election: Requested vote from peers 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047), 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191)
I20260811 03:05:18.805922 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c66a22ff4f420a849d935eedecebd1" candidate_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863bd409cd0d4058bdc16227e559e379"
I20260811 03:05:18.805945 4073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c66a22ff4f420a849d935eedecebd1" candidate_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39af5b9dd1f44f419bb5374224bff222"
I20260811 03:05:18.806135 3912 raft_consensus.cc:3037] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:18.806166 4073 raft_consensus.cc:3037] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:18.808810 3912 raft_consensus.cc:2454] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6552402ca2054fbcb005b9ce1f80c2e5 in term 1.
I20260811 03:05:18.809314 3676 leader_election.cc:302] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6552402ca2054fbcb005b9ce1f80c2e5, 863bd409cd0d4058bdc16227e559e379; no voters:
I20260811 03:05:18.809556 4073 raft_consensus.cc:2454] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6552402ca2054fbcb005b9ce1f80c2e5 in term 1.
I20260811 03:05:18.809706 4174 raft_consensus.cc:2781] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:18.816023 4174 raft_consensus.cc:686] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6552402ca2054fbcb005b9ce1f80c2e5, State: Running, Role: LEADER
I20260811 03:05:18.816715 4174 consensus_queue.cc:237] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } }
I20260811 03:05:18.822428 2715 catalog_manager.cc:5495] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6552402ca2054fbcb005b9ce1f80c2e5 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } health_report { overall_health: UNKNOWN } } }
I20260811 03:05:18.864207 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.858152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.27.109.1:39493)
0811 03:05:18.858366 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.858372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.858391 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:05:18.858492 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.858496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.858613 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.858770 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:05:18.859261 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.860045 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.861576 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.862071 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.862179 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.862841 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:05:18.863496 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:05:18.863625 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.863756 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.863833 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260811 03:05:18.868322 4137 tablet.cc:2369] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:18.873925 3912 raft_consensus.cc:1261] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 1 FOLLOWER]: Refusing update from remote peer 6552402ca2054fbcb005b9ce1f80c2e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:18.873932 4073 raft_consensus.cc:1261] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 1 FOLLOWER]: Refusing update from remote peer 6552402ca2054fbcb005b9ce1f80c2e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:18.875116 4179 consensus_queue.cc:1035] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:18.875536 4174 consensus_queue.cc:1035] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:18.889588 4183 mvcc.cc:205] Tried to move back new op lower bound from 7317166157295636480 to 7317166157077925888. Current Snapshot: MvccSnapshot[applied={T|T < 7317166157295636480}]
I20260811 03:05:18.897693 4184 mvcc.cc:205] Tried to move back new op lower bound from 7317166157295636480 to 7317166157077925888. Current Snapshot: MvccSnapshot[applied={T|T < 7317166157295636480}]
I20260811 03:05:18.906237 4186 mvcc.cc:205] Tried to move back new op lower bound from 7317166157295636480 to 7317166157077925888. Current Snapshot: MvccSnapshot[applied={T|T < 7317166157295636480}]
I20260811 03:05:18.926563 2714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:18.926918 2714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:18.937695 2714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3a23560c8fe4b3091ae72834ae3a9cc: Sending DeleteTablet for 3 replicas of tablet 58c66a22ff4f420a849d935eedecebd1
I20260811 03:05:18.939332 3727 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c66a22ff4f420a849d935eedecebd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:18 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:50408
I20260811 03:05:18.939702 3891 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c66a22ff4f420a849d935eedecebd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:18 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:60972
I20260811 03:05:18.939776 4053 tablet_service.cc:1514] Processing DeleteTablet for tablet 58c66a22ff4f420a849d935eedecebd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:18 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:39222
I20260811 03:05:18.953289 4194 tablet_replica.cc:330] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: stopping tablet replica
I20260811 03:05:18.954067 4194 raft_consensus.cc:2229] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:05:18.955085 4194 raft_consensus.cc:2258] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:18.958408 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 2678
I20260811 03:05:18.959609 4194 ts_tablet_manager.cc:1905] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:18.959698 2750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:18.969185 4196 tablet_replica.cc:330] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: stopping tablet replica
I20260811 03:05:18.969758 4196 raft_consensus.cc:2229] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:18.970186 4194 ts_tablet_manager.cc:1918] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:18.970395 4196 raft_consensus.cc:2258] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:18.970525 4194 log.cc:1196] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/58c66a22ff4f420a849d935eedecebd1
I20260811 03:05:18.971482 4194 ts_tablet_manager.cc:1939] T 58c66a22ff4f420a849d935eedecebd1 P 6552402ca2054fbcb005b9ce1f80c2e5: Deleting consensus metadata
I20260811 03:05:18.973407 4195 tablet_replica.cc:330] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: stopping tablet replica
I20260811 03:05:18.973919 4195 raft_consensus.cc:2229] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:18.974521 4195 raft_consensus.cc:2258] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:18.975303 4196 ts_tablet_manager.cc:1905] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:18.981213 4195 ts_tablet_manager.cc:1905] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:18.981446 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.942390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51596 (local address 127.27.109.61:32823)
0811 03:05:18.942643 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.942654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.958422 (+ 15768us) server_negotiation.cc:407] Connection header received
0811 03:05:18.958648 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.958657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.958795 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.958938 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:18.962206 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.963761 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.969063 (+ 5302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.969822 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.969943 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.974004 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:18.974012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:18.974017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:18.974067 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:18.977839 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.978880 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.978886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.978890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.979043 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.979485 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.979501 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.979504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.979916 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:18.980129 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:18.980822 (+ 693us) server_negotiation.cc:299] Negotiation successful
0811 03:05:18.981092 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 03:05:18.984895 4196 ts_tablet_manager.cc:1918] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:18.986769 4196 log.cc:1196] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/58c66a22ff4f420a849d935eedecebd1
I20260811 03:05:18.987500 4196 ts_tablet_manager.cc:1939] T 58c66a22ff4f420a849d935eedecebd1 P 39af5b9dd1f44f419bb5374224bff222: Deleting consensus metadata
I20260811 03:05:18.997852 4195 ts_tablet_manager.cc:1918] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:18.998062 4195 log.cc:1196] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/58c66a22ff4f420a849d935eedecebd1
I20260811 03:05:18.998759 4195 ts_tablet_manager.cc:1939] T 58c66a22ff4f420a849d935eedecebd1 P 863bd409cd0d4058bdc16227e559e379: Deleting consensus metadata
I20260811 03:05:19.002167 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:18.957608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.27.109.60:42879)
0811 03:05:18.957834 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:18.957841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:18.982517 (+ 24676us) server_negotiation.cc:407] Connection header received
0811 03:05:18.982731 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:18.982739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:18.982838 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:18.982939 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:18.985249 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.986445 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.988585 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:18.989240 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:18.989407 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:18.994242 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:18.994251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:18.994256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:18.994341 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:18.998283 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:18.999390 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:18.999397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:18.999401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:18.999597 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:19.000132 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:19.000138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:19.000141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:19.000572 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:19.000876 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:19.001537 (+ 661us) server_negotiation.cc:299] Negotiation successful
0811 03:05:19.001786 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:05:19.103435 2700 catalog_manager.cc:4841] TS 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047): tablet 58c66a22ff4f420a849d935eedecebd1 (table test-table [id=ade05b6177f0460f81223b0b31b65678]) successfully deleted
I20260811 03:05:19.104336 2697 catalog_manager.cc:4841] TS 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191): tablet 58c66a22ff4f420a849d935eedecebd1 (table test-table [id=ade05b6177f0460f81223b0b31b65678]) successfully deleted
I20260811 03:05:19.104439 2700 catalog_manager.cc:4841] TS 6552402ca2054fbcb005b9ce1f80c2e5 (127.27.109.1:39493): tablet 58c66a22ff4f420a849d935eedecebd1 (table test-table [id=ade05b6177f0460f81223b0b31b65678]) successfully deleted
I20260811 03:05:19.105409 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 2678
I20260811 03:05:19.167894 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:19.147549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.27.109.60:42879)
0811 03:05:19.147773 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:19.147779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:19.148320 (+ 541us) server_negotiation.cc:407] Connection header received
0811 03:05:19.148504 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:19.148509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:19.148584 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:19.148689 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:19.150238 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:19.151299 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:19.153331 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:19.155129 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:19.155231 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:19.159708 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:19.159716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:19.159724 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:19.159792 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:19.163785 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:19.165233 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:19.165239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:19.165243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:19.165401 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:19.165818 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:19.165823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:19.165827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:19.166213 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:19.166448 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:19.167277 (+ 829us) server_negotiation.cc:299] Negotiation successful
0811 03:05:19.167546 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 03:05:19.167932 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:19.146012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.27.109.61:32823)
0811 03:05:19.146242 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:19.146249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:19.147597 (+ 1348us) server_negotiation.cc:407] Connection header received
0811 03:05:19.147940 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:19.147949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:19.148049 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:19.148167 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:19.150857 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:19.151986 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:19.154116 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:19.154884 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:19.154991 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:19.159720 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:19.159728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:19.159737 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:19.159793 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:19.164123 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:19.165248 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:19.165255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:19.165259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:19.165429 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:19.165926 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:19.165932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:19.165937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:19.166325 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:19.166559 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:19.167322 (+ 763us) server_negotiation.cc:299] Negotiation successful
0811 03:05:19.167568 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 03:05:19.904448 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:19.903592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.1:60577)
0811 03:05:19.904104 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:19.904235 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":243,"threads_started":1}
W20260811 03:05:19.905010 3796 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:41275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
I20260811 03:05:19.921657 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:19.920891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.2:40869)
0811 03:05:19.921346 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:19.921484 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
W20260811 03:05:19.922189 3961 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:41275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
I20260811 03:05:19.930855 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:19.929785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.3:51873)
0811 03:05:19.930397 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:19.930536 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":347,"threads_started":1}
W20260811 03:05:19.931264 4127 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:41275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
I20260811 03:05:20.436117 4208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3a23560c8fe4b3091ae72834ae3a9cc)
I20260811 03:05:20.436386 4208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } }
I20260811 03:05:20.437570 4209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3a23560c8fe4b3091ae72834ae3a9cc)
I20260811 03:05:20.437887 4208 leader_election.cc:288] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3a23560c8fe4b3091ae72834ae3a9cc (127.27.109.62:41275), 925210eabbaa41edb648ac0089514df7 (127.27.109.60:42879)
I20260811 03:05:20.437825 4209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } }
I20260811 03:05:20.439079 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.438012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59392)
0811 03:05:20.438595 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.438794 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":282,"threads_started":1}
W20260811 03:05:20.439357 3378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
I20260811 03:05:20.439564 4209 leader_election.cc:288] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3a23560c8fe4b3091ae72834ae3a9cc (127.27.109.62:41275), 00d96f2bcd114aa6a24abaf4352cd7c7 (127.27.109.61:32823)
I20260811 03:05:20.442292 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.441428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59416)
0811 03:05:20.441902 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.442090 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356}
I20260811 03:05:20.442657 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.439151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59406)
0811 03:05:20.441840 (+ 2689us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.442217 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2584,"threads_started":1}
W20260811 03:05:20.442616 3378 leader_election.cc:334] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3a23560c8fe4b3091ae72834ae3a9cc (127.27.109.62:41275): Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
W20260811 03:05:20.443356 3582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
I20260811 03:05:20.445611 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.444969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59428)
0811 03:05:20.445137 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.445352 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 03:05:20.445925 3582 leader_election.cc:334] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3a23560c8fe4b3091ae72834ae3a9cc (127.27.109.62:41275): Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
I20260811 03:05:20.447649 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.438750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:42879 (local address 127.0.0.1:53412)
0811 03:05:20.439296 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.439316 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:20.439476 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:20.440639 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:20.440646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:20.440705 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:20.440962 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.440989 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.443039 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.443047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.445118 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.445134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.447055 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.447060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.447212 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.447238 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 03:05:20.447345 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 03:05:20.447456 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":2019,"threads_started":1}
I20260811 03:05:20.447943 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.438402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.27.109.60:42879)
0811 03:05:20.440109 (+ 1707us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:20.440116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:20.440137 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:05:20.440216 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:20.440224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:20.440313 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:20.440428 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:20.441199 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.442826 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.445292 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.446793 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.446933 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.447048 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:20.447398 (+ 350us) server_negotiation.cc:299] Negotiation successful
0811 03:05:20.447555 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1555,"thread_start_us":1443,"threads_started":1}
I20260811 03:05:20.448534 3607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "925210eabbaa41edb648ac0089514df7" is_pre_election: true
I20260811 03:05:20.448889 3607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d96f2bcd114aa6a24abaf4352cd7c7 in term 1.
I20260811 03:05:20.449488 3379 leader_election.cc:302] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00d96f2bcd114aa6a24abaf4352cd7c7, 925210eabbaa41edb648ac0089514df7; no voters: a3a23560c8fe4b3091ae72834ae3a9cc
I20260811 03:05:20.449939 4215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.439388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:32823 (local address 127.0.0.1:51614)
0811 03:05:20.442913 (+ 3525us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.442935 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:20.443068 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:20.443462 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:20.443467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:20.443548 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:20.444011 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.444028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.445508 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.445514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.447598 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.447612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.449097 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.449103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.449313 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.449354 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 03:05:20.449465 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 03:05:20.449632 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3401,"mutex_wait_us":1599,"thread_start_us":614,"threads_started":1}
I20260811 03:05:20.449946 4208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 03:05:20.450230 4214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.440334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.27.109.61:32823)
0811 03:05:20.441430 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:20.441435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:20.443019 (+ 1584us) server_negotiation.cc:407] Connection header received
0811 03:05:20.443220 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:20.443231 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:20.443300 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:20.443404 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:20.444202 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.445374 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.447977 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.448901 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.449050 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.449176 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:20.449505 (+ 329us) server_negotiation.cc:299] Negotiation successful
0811 03:05:20.449618 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":975,"thread_start_us":794,"threads_started":1}
I20260811 03:05:20.450291 4208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3a23560c8fe4b3091ae72834ae3a9cc)
I20260811 03:05:20.450431 4208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 1 FOLLOWER]: Advancing to term 2
I20260811 03:05:20.451229 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "925210eabbaa41edb648ac0089514df7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" is_pre_election: true
I20260811 03:05:20.452935 4208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } }
I20260811 03:05:20.453411 3403 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 925210eabbaa41edb648ac0089514df7 in current term 2: Already voted for candidate 00d96f2bcd114aa6a24abaf4352cd7c7 in this term.
I20260811 03:05:20.453763 4208 leader_election.cc:288] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [CANDIDATE]: Term 2 election: Requested vote from peers a3a23560c8fe4b3091ae72834ae3a9cc (127.27.109.62:41275), 925210eabbaa41edb648ac0089514df7 (127.27.109.60:42879)
I20260811 03:05:20.454360 3607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "925210eabbaa41edb648ac0089514df7"
I20260811 03:05:20.454653 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.453917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59438)
0811 03:05:20.454265 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.454402 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260811 03:05:20.454728 3607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 1 FOLLOWER]: Advancing to term 2
I20260811 03:05:20.454734 3581 leader_election.cc:302] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 925210eabbaa41edb648ac0089514df7; no voters: 00d96f2bcd114aa6a24abaf4352cd7c7, a3a23560c8fe4b3091ae72834ae3a9cc
I20260811 03:05:20.456122 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.455765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59454)
0811 03:05:20.455925 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.456021 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 03:05:20.456318 3378 leader_election.cc:334] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a3a23560c8fe4b3091ae72834ae3a9cc (127.27.109.62:41275): Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
I20260811 03:05:20.457907 3607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d96f2bcd114aa6a24abaf4352cd7c7 in term 2.
I20260811 03:05:20.458137 4209 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 03:05:20.458523 3379 leader_election.cc:302] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00d96f2bcd114aa6a24abaf4352cd7c7, 925210eabbaa41edb648ac0089514df7; no voters: a3a23560c8fe4b3091ae72834ae3a9cc
I20260811 03:05:20.458918 4208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 03:05:20.459625 4208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [term 2 LEADER]: Becoming Leader. State: Replica: 00d96f2bcd114aa6a24abaf4352cd7c7, State: Running, Role: LEADER
I20260811 03:05:20.460148 4208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } }
I20260811 03:05:20.462361 4218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d96f2bcd114aa6a24abaf4352cd7c7. Latest consensus state: current_term: 2 leader_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:20.462859 4218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:20.463507 4221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:20.467403 4221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:20.468271 4221 catalog_manager.cc:1259] Loaded cluster ID: 64fdd41ffe424b3887d0b354c4dab236
I20260811 03:05:20.468344 4221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:20.469558 4221 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:20.470397 4221 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Loaded TSK: 0
I20260811 03:05:20.471371 4221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:05:20.514003 3393 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:05:20.521610 3393 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:05:20.526644 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.526048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59476)
0811 03:05:20.526283 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.526392 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260811 03:05:20.526612 3607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [term 2 FOLLOWER]: Refusing update from remote peer 00d96f2bcd114aa6a24abaf4352cd7c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260811 03:05:20.527482 4218 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260811 03:05:20.527748 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.527330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59490)
0811 03:05:20.527499 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.527568 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 03:05:20.527933 3378 consensus_peers.cc:487] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 -> Peer a3a23560c8fe4b3091ae72834ae3a9cc (127.27.109.62:41275): Couldn't send request to peer a3a23560c8fe4b3091ae72834ae3a9cc. Status: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 03:05:20.530741 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d96f2bcd114aa6a24abaf4352cd7c7. Latest consensus state: current_term: 2 leader_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:20.531173 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:05:20.532739 4218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:20.533058 4218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:20.534020 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3a23560c8fe4b3091ae72834ae3a9cc" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41275 } } peers { permanent_uuid: "00d96f2bcd114aa6a24abaf4352cd7c7" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 32823 } attrs { promote: false } } peers { permanent_uuid: "925210eabbaa41edb648ac0089514df7" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 42879 } attrs { promote: false } } }
I20260811 03:05:20.534310 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 925210eabbaa41edb648ac0089514df7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:05:20.556865 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.541675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45191 (local address 127.0.0.1:60978)
0811 03:05:20.541883 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.541906 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:20.542029 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:20.543277 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:20.543285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:20.543324 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:20.543646 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.543657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.551503 (+ 7846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.551514 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.554098 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.554122 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.555999 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.556005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.556202 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.556244 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 03:05:20.556320 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 03:05:20.556487 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260811 03:05:20.557934 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.541861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.27.109.2:45191)
0811 03:05:20.542896 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:20.542903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:20.542924 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:05:20.543004 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:20.543011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:20.543107 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:20.543219 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:20.543794 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.551214 (+ 7420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.554371 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.555781 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.557233 (+ 1452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.557349 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:20.557420 (+ 71us) server_negotiation.cc:299] Negotiation successful
0811 03:05:20.557595 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":792,"threads_started":1}
I20260811 03:05:20.559528 3891 tablet_service.cc:1467] Processing CreateTablet for tablet 748e67fb7da84e62a7264fda3b7e1a94 (DEFAULT_TABLE table=test-table [id=6464493f12c743b88f3cf15c2cd74b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:20.560562 3891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748e67fb7da84e62a7264fda3b7e1a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:20.564343 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.542233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:40047 (local address 127.0.0.1:39238)
0811 03:05:20.542412 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.542429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:20.542542 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:20.554791 (+ 12249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:20.554799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:20.554859 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:20.555173 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.555185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.558937 (+ 3752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.558944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.560953 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.560969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.563422 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.563430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.563647 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.563691 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 03:05:20.563770 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 03:05:20.563934 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 03:05:20.565081 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.544322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:39493 (local address 127.0.0.1:50428)
0811 03:05:20.545015 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.545043 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:20.545179 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:20.557936 (+ 12757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:20.557941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:20.557978 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:20.558477 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.558495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.560360 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.560366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.562471 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:20.562490 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.564266 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.564273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:20.564483 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.564518 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 03:05:20.564588 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 03:05:20.564723 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":321,"threads_started":1}
I20260811 03:05:20.565271 4229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.542476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.27.109.3:40047)
0811 03:05:20.554228 (+ 11752us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:20.554237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:20.554257 (+ 20us) server_negotiation.cc:407] Connection header received
0811 03:05:20.554344 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:20.554349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:20.554456 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:20.555338 (+ 882us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:20.555527 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.557723 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.561259 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.563159 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.564631 (+ 1472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.564745 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:20.564804 (+ 59us) server_negotiation.cc:299] Negotiation successful
0811 03:05:20.564963 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11582,"thread_start_us":12086,"threads_started":1}
I20260811 03:05:20.566083 4053 tablet_service.cc:1467] Processing CreateTablet for tablet 748e67fb7da84e62a7264fda3b7e1a94 (DEFAULT_TABLE table=test-table [id=6464493f12c743b88f3cf15c2cd74b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:20.566354 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.544493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.27.109.1:39493)
0811 03:05:20.556997 (+ 12504us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:20.557004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:20.557025 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:05:20.557101 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:20.557109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:20.557437 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:20.558647 (+ 1210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:20.558880 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.560132 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.562742 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.564007 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.565361 (+ 1354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.565825 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:20.565880 (+ 55us) server_negotiation.cc:299] Negotiation successful
0811 03:05:20.566031 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12268,"thread_start_us":12140,"threads_started":1}
I20260811 03:05:20.566627 4053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748e67fb7da84e62a7264fda3b7e1a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:20.570369 4232 tablet_bootstrap.cc:492] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: Bootstrap starting.
I20260811 03:05:20.570749 3727 tablet_service.cc:1467] Processing CreateTablet for tablet 748e67fb7da84e62a7264fda3b7e1a94 (DEFAULT_TABLE table=test-table [id=6464493f12c743b88f3cf15c2cd74b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:20.571228 3727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748e67fb7da84e62a7264fda3b7e1a94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:20.572412 4232 tablet_bootstrap.cc:654] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:20.574810 4232 tablet_bootstrap.cc:492] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: No bootstrap required, opened a new log
I20260811 03:05:20.574994 4232 ts_tablet_manager.cc:1397] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 03:05:20.575850 4232 raft_consensus.cc:348] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.576134 4232 raft_consensus.cc:374] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:20.576251 4232 raft_consensus.cc:729] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863bd409cd0d4058bdc16227e559e379, State: Initialized, Role: FOLLOWER
I20260811 03:05:20.576502 4232 consensus_queue.cc:260] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.587106 4232 ts_tablet_manager.cc:1428] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260811 03:05:20.589884 3962 heartbeater.cc:503] Master 127.27.109.61:32823 was elected leader, sending a full tablet report...
I20260811 03:05:20.591293 4236 tablet_bootstrap.cc:492] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: Bootstrap starting.
I20260811 03:05:20.594170 4236 tablet_bootstrap.cc:654] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:20.594997 4233 tablet_bootstrap.cc:492] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: Bootstrap starting.
I20260811 03:05:20.597409 4233 tablet_bootstrap.cc:654] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:20.599622 4236 tablet_bootstrap.cc:492] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: No bootstrap required, opened a new log
I20260811 03:05:20.599807 4236 ts_tablet_manager.cc:1397] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260811 03:05:20.601047 4233 tablet_bootstrap.cc:492] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: No bootstrap required, opened a new log
I20260811 03:05:20.601218 4233 ts_tablet_manager.cc:1397] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260811 03:05:20.602196 4233 raft_consensus.cc:348] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.602499 4233 raft_consensus.cc:374] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:20.602617 4233 raft_consensus.cc:729] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39af5b9dd1f44f419bb5374224bff222, State: Initialized, Role: FOLLOWER
I20260811 03:05:20.602991 4233 consensus_queue.cc:260] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.603350 4236 raft_consensus.cc:348] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.603670 4236 raft_consensus.cc:374] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:20.603812 4236 raft_consensus.cc:729] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6552402ca2054fbcb005b9ce1f80c2e5, State: Initialized, Role: FOLLOWER
I20260811 03:05:20.604137 4236 consensus_queue.cc:260] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.614439 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.586334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.2:39339)
0811 03:05:20.610273 (+ 23939us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.610507 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23722,"thread_start_us":27518,"threads_started":1}
I20260811 03:05:20.616816 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.615436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.2:51717)
0811 03:05:20.616359 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.616536 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":406,"threads_started":1}
I20260811 03:05:20.616809 4233 ts_tablet_manager.cc:1428] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: Time spent starting tablet: real 0.015s user 0.004s sys 0.012s
I20260811 03:05:20.617475 4236 ts_tablet_manager.cc:1428] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: Time spent starting tablet: real 0.018s user 0.004s sys 0.009s
I20260811 03:05:20.618757 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.617669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.1:46665)
0811 03:05:20.618263 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.618429 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":727,"threads_started":1}
I20260811 03:05:20.621186 3797 heartbeater.cc:503] Master 127.27.109.61:32823 was elected leader, sending a full tablet report...
I20260811 03:05:20.630498 4241 raft_consensus.cc:482] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:20.630851 4241 raft_consensus.cc:504] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.632097 4128 heartbeater.cc:503] Master 127.27.109.61:32823 was elected leader, sending a full tablet report...
I20260811 03:05:20.632087 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748e67fb7da84e62a7264fda3b7e1a94" candidate_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863bd409cd0d4058bdc16227e559e379" is_pre_election: true
I20260811 03:05:20.632354 3912 raft_consensus.cc:2454] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6552402ca2054fbcb005b9ce1f80c2e5 in term 0.
I20260811 03:05:20.632885 3676 leader_election.cc:302] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6552402ca2054fbcb005b9ce1f80c2e5, 863bd409cd0d4058bdc16227e559e379; no voters:
I20260811 03:05:20.633639 4246 raft_consensus.cc:2781] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:20.633752 4246 raft_consensus.cc:482] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:20.633874 4246 raft_consensus.cc:3037] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:20.636432 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.619351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.1:52717)
0811 03:05:20.636154 (+ 16803us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.636260 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16671,"threads_started":1}
I20260811 03:05:20.637384 4246 raft_consensus.cc:504] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.638624 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748e67fb7da84e62a7264fda3b7e1a94" candidate_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863bd409cd0d4058bdc16227e559e379"
I20260811 03:05:20.638938 3912 raft_consensus.cc:3037] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:20.642369 3912 raft_consensus.cc:2454] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6552402ca2054fbcb005b9ce1f80c2e5 in term 1.
I20260811 03:05:20.642916 3676 leader_election.cc:302] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6552402ca2054fbcb005b9ce1f80c2e5, 863bd409cd0d4058bdc16227e559e379; no voters:
I20260811 03:05:20.643720 4247 raft_consensus.cc:2781] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:20.644161 4241 leader_election.cc:288] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191), 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047)
I20260811 03:05:20.643744 4246 leader_election.cc:288] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [CANDIDATE]: Term 1 election: Requested vote from peers 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191), 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047)
I20260811 03:05:20.650578 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.622577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.3:50443)
0811 03:05:20.650173 (+ 27596us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.650315 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27400,"threads_started":1}
I20260811 03:05:20.650821 4247 raft_consensus.cc:686] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6552402ca2054fbcb005b9ce1f80c2e5, State: Running, Role: LEADER
I20260811 03:05:20.651364 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.649731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.1:34599)
0811 03:05:20.650855 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.651101 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":972}
I20260811 03:05:20.651326 4247 consensus_queue.cc:237] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } }
I20260811 03:05:20.651641 4073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748e67fb7da84e62a7264fda3b7e1a94" candidate_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39af5b9dd1f44f419bb5374224bff222" is_pre_election: true
I20260811 03:05:20.651892 4073 raft_consensus.cc:2454] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6552402ca2054fbcb005b9ce1f80c2e5 in term 0.
I20260811 03:05:20.652501 4073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748e67fb7da84e62a7264fda3b7e1a94" candidate_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39af5b9dd1f44f419bb5374224bff222"
I20260811 03:05:20.652683 4073 raft_consensus.cc:3037] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:20.655891 3392 catalog_manager.cc:5495] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6552402ca2054fbcb005b9ce1f80c2e5 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6552402ca2054fbcb005b9ce1f80c2e5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 39493 } health_report { overall_health: HEALTHY } } }
I20260811 03:05:20.658533 4073 raft_consensus.cc:2454] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6552402ca2054fbcb005b9ce1f80c2e5 in term 1.
W20260811 03:05:20.659986 3964 tablet.cc:2369] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:20.661450 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.659167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.3:50275)
0811 03:05:20.661155 (+ 1988us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.661285 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1849}
I20260811 03:05:20.675257 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.669941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.27.109.1:39493)
0811 03:05:20.670175 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:20.670181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:20.670202 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:05:20.670407 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:20.670413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:20.670495 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:20.670603 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:05:20.671032 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.671805 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.673249 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.673782 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.673921 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.674403 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:05:20.674533 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:05:20.674641 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:20.674825 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 03:05:20.674929 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 03:05:20.679040 3912 raft_consensus.cc:1261] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 1 FOLLOWER]: Refusing update from remote peer 6552402ca2054fbcb005b9ce1f80c2e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:20.679080 4073 raft_consensus.cc:1261] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 1 FOLLOWER]: Refusing update from remote peer 6552402ca2054fbcb005b9ce1f80c2e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:20.679786 4248 consensus_queue.cc:1035] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863bd409cd0d4058bdc16227e559e379" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:20.680225 4247 consensus_queue.cc:1035] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39af5b9dd1f44f419bb5374224bff222" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:20.684326 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.683731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.2:60797)
0811 03:05:20.683935 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.684069 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260811 03:05:20.685477 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.684298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.3:37159)
0811 03:05:20.685041 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.685180 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595}
W20260811 03:05:20.688720 3799 tablet.cc:2369] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:20.694307 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.693761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.1:56323)
0811 03:05:20.694026 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.694131 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260811 03:05:20.694356 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.693761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.3:51617)
0811 03:05:20.694126 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.694208 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20260811 03:05:20.699868 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.699226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.27.109.2:54053)
0811 03:05:20.699535 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:20.699665 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260811 03:05:20.705394 3392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:20.705828 3392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:20.713086 3392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00d96f2bcd114aa6a24abaf4352cd7c7: Sending DeleteTablet for 3 replicas of tablet 748e67fb7da84e62a7264fda3b7e1a94
I20260811 03:05:20.714246 3891 tablet_service.cc:1514] Processing DeleteTablet for tablet 748e67fb7da84e62a7264fda3b7e1a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:20 UTC) from {username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:60978
I20260811 03:05:20.714916 4053 tablet_service.cc:1514] Processing DeleteTablet for tablet 748e67fb7da84e62a7264fda3b7e1a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:20 UTC) from {username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:39238
I20260811 03:05:20.714955 3727 tablet_service.cc:1514] Processing DeleteTablet for tablet 748e67fb7da84e62a7264fda3b7e1a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:20 UTC) from {username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:50428
I20260811 03:05:20.716759 4052 tablet_service.cc:1514] Processing DeleteTablet for tablet 748e67fb7da84e62a7264fda3b7e1a94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:20 UTC) from {username='oryx', principal='oryx/127.27.109.61@KRBTEST.COM'} at 127.0.0.1:39238
I20260811 03:05:20.717469 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 3577
I20260811 03:05:20.727849 4264 tablet_replica.cc:330] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: stopping tablet replica
I20260811 03:05:20.728361 4264 raft_consensus.cc:2229] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:05:20.729045 4264 raft_consensus.cc:2258] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:20.729166 4263 tablet_replica.cc:330] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: stopping tablet replica
I20260811 03:05:20.729387 4263 raft_consensus.cc:2229] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:20.729737 4263 raft_consensus.cc:2258] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:20.731127 4264 ts_tablet_manager.cc:1905] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:20.732039 4263 ts_tablet_manager.cc:1905] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:20.738198 4260 tablet_replica.cc:330] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: stopping tablet replica
I20260811 03:05:20.738440 4260 raft_consensus.cc:2229] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:20.739542 4264 ts_tablet_manager.cc:1918] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:20.739831 4264 log.cc:1196] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/748e67fb7da84e62a7264fda3b7e1a94
I20260811 03:05:20.740428 4264 ts_tablet_manager.cc:1939] T 748e67fb7da84e62a7264fda3b7e1a94 P 6552402ca2054fbcb005b9ce1f80c2e5: Deleting consensus metadata
I20260811 03:05:20.741925 3379 catalog_manager.cc:4841] TS 6552402ca2054fbcb005b9ce1f80c2e5 (127.27.109.1:39493): tablet 748e67fb7da84e62a7264fda3b7e1a94 (table test-table [id=6464493f12c743b88f3cf15c2cd74b78]) successfully deleted
I20260811 03:05:20.742363 4260 raft_consensus.cc:2258] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:20.743873 4260 ts_tablet_manager.cc:1905] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:20.743942 4263 ts_tablet_manager.cc:1918] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:20.744228 4263 log.cc:1196] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/748e67fb7da84e62a7264fda3b7e1a94
I20260811 03:05:20.744764 4263 ts_tablet_manager.cc:1939] T 748e67fb7da84e62a7264fda3b7e1a94 P 39af5b9dd1f44f419bb5374224bff222: Deleting consensus metadata
I20260811 03:05:20.746291 3379 catalog_manager.cc:4841] TS 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047): tablet 748e67fb7da84e62a7264fda3b7e1a94 (table test-table [id=6464493f12c743b88f3cf15c2cd74b78]) successfully deleted
W20260811 03:05:20.747008 3379 catalog_manager.cc:4808] TS 39af5b9dd1f44f419bb5374224bff222 (127.27.109.3:40047): delete failed for tablet 748e67fb7da84e62a7264fda3b7e1a94 because the tablet was not found. No further retry: Not found: Tablet not found: 748e67fb7da84e62a7264fda3b7e1a94
I20260811 03:05:20.749583 4260 ts_tablet_manager.cc:1918] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:20.749758 4260 log.cc:1196] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/748e67fb7da84e62a7264fda3b7e1a94
I20260811 03:05:20.750169 4260 ts_tablet_manager.cc:1939] T 748e67fb7da84e62a7264fda3b7e1a94 P 863bd409cd0d4058bdc16227e559e379: Deleting consensus metadata
I20260811 03:05:20.751446 3377 catalog_manager.cc:4841] TS 863bd409cd0d4058bdc16227e559e379 (127.27.109.2:45191): tablet 748e67fb7da84e62a7264fda3b7e1a94 (table test-table [id=6464493f12c743b88f3cf15c2cd74b78]) successfully deleted
I20260811 03:05:20.753618 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.718871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.27.109.1:39493)
0811 03:05:20.719139 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:20.719149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:20.729254 (+ 10105us) server_negotiation.cc:407] Connection header received
0811 03:05:20.729907 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:20.729919 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:20.730032 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:20.730122 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:20.732759 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.734160 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.736629 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.737656 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.737789 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.743026 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:20.743056 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:20.743082 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:20.743142 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:20.749592 (+ 6450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:20.750945 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:20.750956 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:20.750960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:20.751153 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:20.751841 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:20.751846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:20.751848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:20.752251 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:20.752501 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:20.753071 (+ 570us) server_negotiation.cc:299] Negotiation successful
0811 03:05:20.753293 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:05:20.754437 3790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:20.883136 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 3577
I20260811 03:05:20.916491 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 3810
I20260811 03:05:20.932790 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:20.917393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.27.109.2:45191)
0811 03:05:20.917629 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:20.917636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:20.917665 (+ 29us) server_negotiation.cc:407] Connection header received
0811 03:05:20.917870 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:20.917876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:20.917958 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:20.918080 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:20.919648 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.920778 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.922053 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:20.922830 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:20.922954 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:20.926024 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:20.926047 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:20.926069 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:20.926112 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:20.929779 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:20.930554 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:20.930562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:20.930565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:20.930778 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:20.931247 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:20.931252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:20.931254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:20.931675 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:20.931985 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:20.932345 (+ 360us) server_negotiation.cc:299] Negotiation successful
0811 03:05:20.932537 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 03:05:20.933570 3954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:21.033339 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 3810
I20260811 03:05:21.058193 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 3980
I20260811 03:05:21.073657 4229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:21.059200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.27.109.3:40047)
0811 03:05:21.059445 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:21.059454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:21.059477 (+ 23us) server_negotiation.cc:407] Connection header received
0811 03:05:21.059717 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:21.059722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:21.059791 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:21.060036 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:21.061486 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:21.062257 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:21.063603 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:21.064033 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:21.064187 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:21.067239 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:21.067258 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:21.067276 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:21.067320 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:21.070345 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:21.071423 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:21.071440 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:21.071442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:21.071590 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:21.072009 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:21.072014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:21.072016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:21.072414 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:21.072744 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:21.073074 (+ 330us) server_negotiation.cc:299] Negotiation successful
0811 03:05:21.073327 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 03:05:21.074411 4120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:21.120226 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:21.119312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59494)
0811 03:05:21.119948 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:21.120046 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":309,"threads_started":1}
I20260811 03:05:21.121765 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:21.121358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41275 (local address 127.0.0.1:59510)
0811 03:05:21.121531 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:21.121627 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260811 03:05:21.175038 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 3980
I20260811 03:05:21.197376 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 3234
I20260811 03:05:21.198802 3423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:21.331099 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 3234
I20260811 03:05:21.360006 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 3353
I20260811 03:05:21.361263 3629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:21.487452 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 3353
2026-08-11T03:05:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10596 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:05:21 dist-test-slave-8tdl krb5kdc[4275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:05:21 dist-test-slave-8tdl krb5kdc[4275](info): set up 2 sockets
Aug 11 03:05:21 dist-test-slave-8tdl krb5kdc[4275](info): commencing operation
krb5kdc: starting...
W20260811 03:05:22.711537 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:05:22 dist-test-slave-8tdl krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.62.keytab.
Entry for principal oryx/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.62.keytab.
I20260811 03:05:23.011523 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39269
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.62.keytab
--principal=oryx/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:23.367075 4291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:23.367419 4291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:23.367514 4291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:23.375602 4291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:23.375744 4291 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 03:05:23.375787 4291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:23.375852 4291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:23.375908 4291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:23.375973 4291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:23.386507 4291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.62.keytab
--principal=oryx/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:23.387931 4291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:23.389272 4291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:23.402767 4296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:23.402798 4299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:23.402735 4297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:23.403807 4291 server_base.cc:1034] running on GCE node
I20260811 03:05:23.404742 4291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:23.406373 4291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:23.407677 4291 hybrid_clock.cc:648] HybridClock initialized: now 1786417523407590 us; error 61 us; skew 500 ppm
I20260811 03:05:23.408120 4291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:23 dist-test-slave-8tdl krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417523, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:23.416844 4291 init.cc:377] Logged in from keytab as oryx/127.27.109.62@KRBTEST.COM (short username oryx)
I20260811 03:05:23.418643 4291 webserver.cc:460] Webserver started at http://127.27.109.62:41209/ using document root <none> and password file <none>
I20260811 03:05:23.419499 4291 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:23.419643 4291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:23.420019 4291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:23.423422 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b"
format_stamp: "Formatted at 2026-08-11 03:05:23 on dist-test-slave-8tdl"
server_key: "7572a92049d1e4cccd447d59a50dd7ac"
server_key_iv: "acd1c5fea1d0d9de745507c4a3cbb831"
server_key_version: "encryptionkey@0"
I20260811 03:05:23.424280 4291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b"
format_stamp: "Formatted at 2026-08-11 03:05:23 on dist-test-slave-8tdl"
server_key: "7572a92049d1e4cccd447d59a50dd7ac"
server_key_iv: "acd1c5fea1d0d9de745507c4a3cbb831"
server_key_version: "encryptionkey@0"
I20260811 03:05:23.430007 4291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:05:23.434124 4308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:23.435634 4291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:23.435842 4291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b"
format_stamp: "Formatted at 2026-08-11 03:05:23 on dist-test-slave-8tdl"
server_key: "7572a92049d1e4cccd447d59a50dd7ac"
server_key_iv: "acd1c5fea1d0d9de745507c4a3cbb831"
server_key_version: "encryptionkey@0"
I20260811 03:05:23.436111 4291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:23.453596 4291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:23.458397 4291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:23.458789 4291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:23.484548 4291 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39269
I20260811 03:05:23.484571 4375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39269 every 8 connection(s)
I20260811 03:05:23.486116 4291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:23.490211 4376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:23.492828 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 4291
I20260811 03:05:23.493086 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:23.493917 28084 external_mini_cluster.cc:1428] Setting key 5f58830a63fbcee6e76e57738f27fd86
I20260811 03:05:23.504125 4376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: Bootstrap starting.
I20260811 03:05:23.508280 4376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:23.509804 4376 log.cc:824] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: Log is configured to *not* fsync() on all Append() calls
Aug 11 03:05:23 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:23.513226 4376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: No bootstrap required, opened a new log
I20260811 03:05:23.519822 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:23.496236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.27.109.62:39269)
0811 03:05:23.497121 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:23.497133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:23.497178 (+ 45us) server_negotiation.cc:407] Connection header received
0811 03:05:23.499067 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:23.499095 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:23.499523 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:23.499910 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:23.501900 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:23.503367 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:23.505357 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:23.506241 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:23.506428 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:23.511746 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:23.511754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:23.511770 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:23.511810 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:23.515208 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:23.516062 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:23.516070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:23.516076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:23.516274 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:23.516713 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:23.516720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:23.516728 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:23.517325 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:23.517691 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:23.518139 (+ 448us) server_negotiation.cc:299] Negotiation successful
0811 03:05:23.518621 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":412,"threads_started":1}
I20260811 03:05:23.521802 4376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39269 } }
I20260811 03:05:23.522246 4376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:23.522390 4376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f4e85b7f2149cfa82b3fc58bafbf8b, State: Initialized, Role: FOLLOWER
I20260811 03:05:23.523118 4376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39269 } }
I20260811 03:05:23.523407 4376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:23.523566 4376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:23.523749 4376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:23.526053 4376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39269 } }
I20260811 03:05:23.526651 4376 leader_election.cc:302] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03f4e85b7f2149cfa82b3fc58bafbf8b; no voters:
I20260811 03:05:23.527350 4376 leader_election.cc:288] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:23.527602 4381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:23.528395 4381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [term 1 LEADER]: Becoming Leader. State: Replica: 03f4e85b7f2149cfa82b3fc58bafbf8b, State: Running, Role: LEADER
I20260811 03:05:23.529117 4381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39269 } }
I20260811 03:05:23.529881 4376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:23.532042 4382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39269 } } }
I20260811 03:05:23.532397 4382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:23.532630 4383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03f4e85b7f2149cfa82b3fc58bafbf8b. Latest consensus state: current_term: 1 leader_uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03f4e85b7f2149cfa82b3fc58bafbf8b" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39269 } } }
I20260811 03:05:23.533084 4383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:23.537264 4387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:23.543080 4387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:23.552542 4387 catalog_manager.cc:1347] Generated new cluster ID: c804d87f27ad407aba0eeef91a840c66
I20260811 03:05:23.552685 4387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:23.570045 4387 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:23.572359 4387 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:23.584770 4387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: Generated new TSK 0
I20260811 03:05:23.585997 4387 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
I20260811 03:05:23.659308 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39269
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:24.029810 4403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:24.030118 4403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:24.030277 4403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:24.037897 4403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:24.038029 4403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:24.038170 4403 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:05:24.038317 4403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:05:24.048879 4403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:39269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:24.050318 4403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:24.051834 4403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:24.065992 4409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:24.065902 4412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:24.065948 4410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:24.067550 4403 server_base.cc:1034] running on GCE node
I20260811 03:05:24.068241 4403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:24.069366 4403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:24.070648 4403 hybrid_clock.cc:648] HybridClock initialized: now 1786417524070569 us; error 40 us; skew 500 ppm
I20260811 03:05:24.071069 4403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:24 dist-test-slave-8tdl krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417524, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:24.074746 4403 init.cc:377] Logged in from keytab as oryx/127.27.109.1@KRBTEST.COM (short username oryx)
I20260811 03:05:24.076427 4403 webserver.cc:460] Webserver started at http://127.27.109.1:36063/ using document root <none> and password file <none>
I20260811 03:05:24.077267 4403 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:24.077423 4403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:24.077821 4403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:24.081346 4403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "605bc85fec22464d8875471d34bad680"
format_stamp: "Formatted at 2026-08-11 03:05:24 on dist-test-slave-8tdl"
server_key: "86bae079c416d688fba55a95c448eb11"
server_key_iv: "3aeb4f38e3813220d37a147177dffc06"
server_key_version: "encryptionkey@0"
I20260811 03:05:24.082291 4403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "605bc85fec22464d8875471d34bad680"
format_stamp: "Formatted at 2026-08-11 03:05:24 on dist-test-slave-8tdl"
server_key: "86bae079c416d688fba55a95c448eb11"
server_key_iv: "3aeb4f38e3813220d37a147177dffc06"
server_key_version: "encryptionkey@0"
I20260811 03:05:24.088702 4403 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260811 03:05:24.093076 4419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:24.094559 4403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:05:24.094825 4403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "605bc85fec22464d8875471d34bad680"
format_stamp: "Formatted at 2026-08-11 03:05:24 on dist-test-slave-8tdl"
server_key: "86bae079c416d688fba55a95c448eb11"
server_key_iv: "3aeb4f38e3813220d37a147177dffc06"
server_key_version: "encryptionkey@0"
I20260811 03:05:24.095106 4403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:24.121876 4403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:24.127333 4403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:24.127676 4403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:24.131165 4403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:24.131299 4403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:24.131476 4403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:24.131580 4403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:05:24 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417524, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:24.159138 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:24.131443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:35213 (local address 127.27.109.62:39269)
0811 03:05:24.132047 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:24.132058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:24.133674 (+ 1616us) server_negotiation.cc:407] Connection header received
0811 03:05:24.137216 (+ 3542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:24.137223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:24.137331 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:24.137531 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:24.140385 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.141953 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.144646 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.145550 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.145734 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:24.149886 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:24.149898 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:24.149907 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:24.149971 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:24.153963 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:24.155266 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:24.155274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:24.155278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:24.155474 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:24.156213 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:24.156218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:24.156228 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:24.156679 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:24.156847 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:24.158080 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0811 03:05:24.158475 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":278,"threads_started":1}
I20260811 03:05:24.159802 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:24.132332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39269 (local address 127.27.109.1:35213)
0811 03:05:24.133449 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:24.133505 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:24.136737 (+ 3232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:24.138036 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:24.138052 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:24.139065 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:24.140119 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:24.140166 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.142182 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.142193 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:24.144463 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:24.144478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.145871 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.145878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:24.146093 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:24.146869 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:24.146894 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:24.149565 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:24.154137 (+ 4572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:24.154148 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:24.154171 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:24.155007 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:24.155706 (+ 699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:24.155712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:24.155716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:24.156061 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:24.156811 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:24.156819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:24.157533 (+ 714us) client_negotiation.cc:769] Sending connection context
0811 03:05:24.157900 (+ 367us) client_negotiation.cc:240] Negotiation successful
0811 03:05:24.158464 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":445,"threads_started":1}
I20260811 03:05:24.185348 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:24.166416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39269 (local address 127.27.109.1:57957)
0811 03:05:24.167424 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:24.167444 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:24.167629 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:24.168076 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:24.168086 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:24.169010 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:24.169597 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:24.169618 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.170791 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.170801 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:24.173339 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:24.173375 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.174324 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.174335 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:24.174656 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:24.175708 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:24.175752 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:24.177548 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:24.181818 (+ 4270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:24.181828 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:24.181836 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:24.182659 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:24.183261 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:24.183266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:24.183270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:24.183500 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:24.184382 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:24.184387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:24.184590 (+ 203us) client_negotiation.cc:769] Sending connection context
0811 03:05:24.184703 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 03:05:24.184990 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":576,"threads_started":1}
I20260811 03:05:24.186128 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:24.166307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57957 (local address 127.27.109.62:39269)
0811 03:05:24.166489 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:24.166494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:24.167551 (+ 1057us) server_negotiation.cc:407] Connection header received
0811 03:05:24.167761 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:24.167769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:24.167874 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:24.167996 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:24.169823 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.170591 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.173577 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.174060 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.174172 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:24.177828 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:24.177838 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:24.177845 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:24.177903 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:24.181638 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:24.182861 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:24.182869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:24.182875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:24.183128 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:24.183666 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:24.183674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:24.183680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:24.184200 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:24.184399 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:24.184932 (+ 533us) server_negotiation.cc:299] Negotiation successful
0811 03:05:24.185291 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 03:05:24.198855 4403 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:41497
I20260811 03:05:24.199052 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:41497 every 8 connection(s)
I20260811 03:05:24.200336 4403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:05:24.201916 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 4403
I20260811 03:05:24.202206 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:05:24.203091 4554 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39269
I20260811 03:05:24.203167 28084 external_mini_cluster.cc:1428] Setting key ac90ca53ee3cfca2d18f70bfee62c13b
I20260811 03:05:24.203359 4554 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:24.204104 4554 heartbeater.cc:511] Master 127.27.109.62:39269 requested a full tablet report, sending...
I20260811 03:05:24.206442 4328 ts_manager.cc:194] Registered new tserver with Master: 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497)
I20260811 03:05:24.208211 4328 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:35213
WARNING: no policy specified for oryx/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
I20260811 03:05:24.268668 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39269
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:24.634220 4558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:24.634508 4558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:24.634699 4558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:24.642411 4558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:24.642539 4558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:24.642712 4558 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:05:24.642846 4558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:24.653141 4558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:39269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:24.654604 4558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:24.656129 4558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:24.670171 4563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:24.670220 4564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:24.670262 4566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:24.671478 4558 server_base.cc:1034] running on GCE node
I20260811 03:05:24.672194 4558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:24.673274 4558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:24.674548 4558 hybrid_clock.cc:648] HybridClock initialized: now 1786417524674484 us; error 41 us; skew 500 ppm
I20260811 03:05:24.674923 4558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:24 dist-test-slave-8tdl krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417524, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:24.678331 4558 init.cc:377] Logged in from keytab as oryx/127.27.109.2@KRBTEST.COM (short username oryx)
I20260811 03:05:24.680157 4558 webserver.cc:460] Webserver started at http://127.27.109.2:45913/ using document root <none> and password file <none>
I20260811 03:05:24.681012 4558 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:24.681157 4558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:24.681501 4558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:24.684646 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "13949c5b0e7845b988e01a3abf47c0d8"
format_stamp: "Formatted at 2026-08-11 03:05:24 on dist-test-slave-8tdl"
server_key: "c8021591e22074def61fa8cd64f854d7"
server_key_iv: "4023900af7c4f59e2c8c12f23c20008b"
server_key_version: "encryptionkey@0"
I20260811 03:05:24.685557 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "13949c5b0e7845b988e01a3abf47c0d8"
format_stamp: "Formatted at 2026-08-11 03:05:24 on dist-test-slave-8tdl"
server_key: "c8021591e22074def61fa8cd64f854d7"
server_key_iv: "4023900af7c4f59e2c8c12f23c20008b"
server_key_version: "encryptionkey@0"
I20260811 03:05:24.691270 4558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 03:05:24.695329 4573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:24.696753 4558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:05:24.696957 4558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "13949c5b0e7845b988e01a3abf47c0d8"
format_stamp: "Formatted at 2026-08-11 03:05:24 on dist-test-slave-8tdl"
server_key: "c8021591e22074def61fa8cd64f854d7"
server_key_iv: "4023900af7c4f59e2c8c12f23c20008b"
server_key_version: "encryptionkey@0"
I20260811 03:05:24.697235 4558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:24.718991 4558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:24.723397 4558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:24.723721 4558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:24.726902 4558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:24.727025 4558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:24.727255 4558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:24.727330 4558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:05:24 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417524, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:24.755923 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:24.728189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39269 (local address 127.27.109.2:34953)
0811 03:05:24.729080 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:24.729127 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:24.731403 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:24.732231 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:24.732249 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:24.733231 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:24.734287 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:24.734315 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.735960 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.735966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:24.738347 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:24.738366 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.739980 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.739992 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:24.740236 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:24.741240 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:24.741284 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:24.745123 (+ 3839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:24.749783 (+ 4660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:24.749795 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:24.749815 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:24.750389 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:24.750929 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:24.750938 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:24.750943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:24.751368 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:24.752193 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:24.752202 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:24.752885 (+ 683us) client_negotiation.cc:769] Sending connection context
0811 03:05:24.753336 (+ 451us) client_negotiation.cc:240] Negotiation successful
0811 03:05:24.754097 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":398,"threads_started":1}
I20260811 03:05:24.757121 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:24.727438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34953 (local address 127.27.109.62:39269)
0811 03:05:24.728412 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:24.728419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:24.729263 (+ 844us) server_negotiation.cc:407] Connection header received
0811 03:05:24.731701 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:24.731707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:24.731801 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:24.731935 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:24.734508 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.735772 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.738724 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.739648 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.739867 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:24.745482 (+ 5615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:24.745492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:24.745498 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:24.745553 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:24.749601 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:24.750543 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:24.750551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:24.750557 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:24.750790 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:24.751577 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:24.751583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:24.751587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:24.752045 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:24.752217 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:24.756600 (+ 4383us) server_negotiation.cc:299] Negotiation successful
0811 03:05:24.756819 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":713,"threads_started":1}
I20260811 03:05:24.781852 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:24.764780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39269 (local address 127.27.109.2:33257)
0811 03:05:24.765468 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:24.765487 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:24.765685 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:24.766071 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:24.766076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:24.766631 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:24.767208 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:24.767230 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.769205 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.769214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:24.770812 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:24.770833 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.771726 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.771733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:24.771936 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:24.772706 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:24.772725 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:24.773569 (+ 844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:24.777838 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:24.777868 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:24.777873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:24.778800 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:24.779352 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:24.779358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:24.779363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:24.779595 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:24.780586 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:24.780590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:24.781032 (+ 442us) client_negotiation.cc:769] Sending connection context
0811 03:05:24.781168 (+ 136us) client_negotiation.cc:240] Negotiation successful
0811 03:05:24.781481 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":375,"threads_started":1}
I20260811 03:05:24.782207 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:24.764663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:33257 (local address 127.27.109.62:39269)
0811 03:05:24.764852 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:24.764857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:24.765600 (+ 743us) server_negotiation.cc:407] Connection header received
0811 03:05:24.765834 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:24.765844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:24.765918 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:24.766043 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:24.767440 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.768979 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.771012 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:24.771500 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:24.771671 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:24.773810 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:24.773815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:24.773819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:24.773855 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:24.777667 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:24.778958 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:24.778967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:24.778974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:24.779193 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:24.779772 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:24.779781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:24.779786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:24.780410 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:24.780619 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:24.781355 (+ 736us) server_negotiation.cc:299] Negotiation successful
0811 03:05:24.781694 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 03:05:24.785472 4558 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:37217
I20260811 03:05:24.785953 4708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:37217 every 8 connection(s)
I20260811 03:05:24.787241 4558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:24.789824 4709 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39269
I20260811 03:05:24.790006 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 4558
I20260811 03:05:24.790134 4709 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:24.790236 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:05:24.790874 28084 external_mini_cluster.cc:1428] Setting key e2283fbbc80a5ef4dc3582e74ed27efd
I20260811 03:05:24.791131 4709 heartbeater.cc:511] Master 127.27.109.62:39269 requested a full tablet report, sending...
I20260811 03:05:24.792936 4327 ts_manager.cc:194] Registered new tserver with Master: 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217)
I20260811 03:05:24.793977 4327 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:34953
WARNING: no policy specified for oryx/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
I20260811 03:05:24.855116 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39269
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:05:25.211572 4554 heartbeater.cc:503] Master 127.27.109.62:39269 was elected leader, sending a full tablet report...
W20260811 03:05:25.214797 4713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:25.215186 4713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:25.215351 4713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:25.222913 4713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:25.223093 4713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:25.223251 4713 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 03:05:25.223400 4713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:25.233793 4713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.109.62:39269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:25.235270 4713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:25.236701 4713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:25.251336 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:25.251364 4723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:25.252351 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:25.253140 4713 server_base.cc:1034] running on GCE node
I20260811 03:05:25.253927 4713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:25.255081 4713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:25.256342 4713 hybrid_clock.cc:648] HybridClock initialized: now 1786417525256276 us; error 40 us; skew 500 ppm
I20260811 03:05:25.256750 4713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:25 dist-test-slave-8tdl krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417525, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:25.260385 4713 init.cc:377] Logged in from keytab as oryx/127.27.109.3@KRBTEST.COM (short username oryx)
I20260811 03:05:25.262164 4713 webserver.cc:460] Webserver started at http://127.27.109.3:37783/ using document root <none> and password file <none>
I20260811 03:05:25.263043 4713 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:25.263181 4713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:25.263523 4713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:25.266798 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "3025f79396c740baa843ac83250eb65d"
format_stamp: "Formatted at 2026-08-11 03:05:25 on dist-test-slave-8tdl"
server_key: "ef833578157413d858467b053d211e26"
server_key_iv: "a5fad7f1acbd7aa6959cc7f931a719d0"
server_key_version: "encryptionkey@0"
I20260811 03:05:25.267689 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "3025f79396c740baa843ac83250eb65d"
format_stamp: "Formatted at 2026-08-11 03:05:25 on dist-test-slave-8tdl"
server_key: "ef833578157413d858467b053d211e26"
server_key_iv: "a5fad7f1acbd7aa6959cc7f931a719d0"
server_key_version: "encryptionkey@0"
I20260811 03:05:25.273168 4713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 03:05:25.276919 4730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:25.278373 4713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:05:25.278560 4713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "3025f79396c740baa843ac83250eb65d"
format_stamp: "Formatted at 2026-08-11 03:05:25 on dist-test-slave-8tdl"
server_key: "ef833578157413d858467b053d211e26"
server_key_iv: "a5fad7f1acbd7aa6959cc7f931a719d0"
server_key_version: "encryptionkey@0"
I20260811 03:05:25.278891 4713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:25.311719 4713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:25.316414 4713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:25.316738 4713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:25.319845 4713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:25.319967 4713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:25.320151 4713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:25.320231 4713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:05:25 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417525, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:25.349242 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:25.320576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:33845 (local address 127.27.109.62:39269)
0811 03:05:25.321256 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:25.321263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:25.322646 (+ 1383us) server_negotiation.cc:407] Connection header received
0811 03:05:25.326051 (+ 3405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:25.326059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:25.326177 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:25.326293 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:25.329848 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.331149 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.333599 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.334268 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.334421 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:25.340713 (+ 6292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:25.340723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:25.340731 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:25.340790 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:25.344732 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:25.345916 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:25.345922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:25.345926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:25.346088 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:25.346623 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:25.346628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:25.346632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:25.346995 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:25.347170 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:25.348228 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0811 03:05:25.348628 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":425,"threads_started":1}
I20260811 03:05:25.350330 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:25.321402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39269 (local address 127.27.109.3:33845)
0811 03:05:25.322419 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:25.322490 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:25.325554 (+ 3064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:25.326999 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:25.327020 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:25.328282 (+ 1262us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:25.329558 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:25.329595 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.331363 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.331371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:25.333413 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:25.333429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.334534 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.334542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:25.334983 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:25.336052 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:25.336078 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:25.340351 (+ 4273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:25.344929 (+ 4578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:25.344939 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:25.344966 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:25.345759 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:25.346218 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:25.346224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:25.346228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:25.346482 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:25.347146 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:25.347155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:25.347701 (+ 546us) client_negotiation.cc:769] Sending connection context
0811 03:05:25.348119 (+ 418us) client_negotiation.cc:240] Negotiation successful
0811 03:05:25.348767 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":400,"threads_started":1}
I20260811 03:05:25.376879 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:25.358341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:54929 (local address 127.27.109.62:39269)
0811 03:05:25.358620 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:25.358630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:25.359413 (+ 783us) server_negotiation.cc:407] Connection header received
0811 03:05:25.359686 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:25.359695 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:25.359810 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:25.359952 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:25.361267 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.362847 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.365465 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.366050 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.366216 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:25.369207 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:25.369229 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:25.369236 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:25.369294 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:25.373411 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:25.374379 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:25.374386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:25.374391 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:25.374584 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:25.375226 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:25.375230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:25.375233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:25.375580 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:25.375732 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:25.376406 (+ 674us) server_negotiation.cc:299] Negotiation successful
0811 03:05:25.376609 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:05:25.377472 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:25.357819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39269 (local address 127.27.109.3:54929)
0811 03:05:25.359264 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:25.359290 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:25.359533 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:25.360031 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:25.360035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:25.360684 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:25.361093 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:25.361105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.363085 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.363094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:25.365224 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:25.365251 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.366284 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.366291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:25.366501 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:25.367314 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:25.367348 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:25.368894 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:25.373567 (+ 4673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:25.373577 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:25.373581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:25.374199 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:25.374772 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:25.374785 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:25.374792 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:25.375041 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:25.375732 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:25.375740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:25.376021 (+ 281us) client_negotiation.cc:769] Sending connection context
0811 03:05:25.376231 (+ 210us) client_negotiation.cc:240] Negotiation successful
0811 03:05:25.376758 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"thread_start_us":1108,"threads_started":1}
I20260811 03:05:25.378547 4713 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:33703
I20260811 03:05:25.378554 4862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:33703 every 8 connection(s)
I20260811 03:05:25.380340 4713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:25.382877 4863 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39269
I20260811 03:05:25.383124 4863 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:25.383836 4863 heartbeater.cc:511] Master 127.27.109.62:39269 requested a full tablet report, sending...
I20260811 03:05:25.385265 4327 ts_manager.cc:194] Registered new tserver with Master: 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:25.386088 4327 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:33845
I20260811 03:05:25.386826 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 4713
I20260811 03:05:25.387045 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:05:25.387481 28084 external_mini_cluster.cc:1428] Setting key c5a91f523f5e39f2726c512f170b340c
I20260811 03:05:25.390489 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 03:05:25 dist-test-slave-8tdl krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 03:05:25 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:25 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:25.797925 4709 heartbeater.cc:503] Master 127.27.109.62:39269 was elected leader, sending a full tablet report...
I20260811 03:05:26.389029 4863 heartbeater.cc:503] Master 127.27.109.62:39269 was elected leader, sending a full tablet report...
W20260811 03:05:26.590837 4872 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.140s user 0.007s sys 0.014s
W20260811 03:05:26.592238 4739 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:25.444967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.27.109.62:39269)
0811 03:05:25.445184 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:25.445190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:25.445448 (+ 258us) server_negotiation.cc:407] Connection header received
0811 03:05:25.445629 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:25.445634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:25.445723 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:25.445813 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:25.447369 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.448236 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.449693 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:25.450122 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:25.450225 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.591858 (+1141633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56916: BlockingRecv error: recv got EOF from 127.0.0.1:56916 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 11 03:05:26 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417525, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:26.611513 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.596798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.27.109.62:39269)
0811 03:05:26.596994 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:26.596999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:26.597492 (+ 493us) server_negotiation.cc:407] Connection header received
0811 03:05:26.597651 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:26.597656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:26.597722 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:26.597815 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:26.599442 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.600280 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.601809 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.602260 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.602369 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.606358 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:26.606364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:26.606367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:26.606404 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:26.609172 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:26.609846 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:26.609849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:26.609852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:26.609974 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:26.610298 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:26.610301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:26.610303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:26.610575 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:26.610846 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:26.611090 (+ 244us) server_negotiation.cc:299] Negotiation successful
0811 03:05:26.611277 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 03:05:26.615852 4327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:05:26.623319 4327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:05:26.651124 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.636782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41497 (local address 127.0.0.1:45636)
0811 03:05:26.638441 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:26.638481 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:26.638730 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:26.639551 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:26.639562 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:26.639599 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:26.640128 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.640141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.642947 (+ 2806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.642956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.646553 (+ 3597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.646578 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.649309 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.649320 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.649627 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.649720 (+ 93us) client_negotiation.cc:769] Sending connection context
0811 03:05:26.649883 (+ 163us) client_negotiation.cc:240] Negotiation successful
0811 03:05:26.650191 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1364,"mutex_wait_us":80,"thread_start_us":1189,"threads_started":1}
I20260811 03:05:26.651331 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.636878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37217 (local address 127.0.0.1:56388)
0811 03:05:26.639341 (+ 2463us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:26.639356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:26.639504 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:26.640436 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:26.640443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:26.640479 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:26.641235 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.641259 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.643896 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.643904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.647066 (+ 3162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.647089 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.650154 (+ 3065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.650163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.650469 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.650527 (+ 58us) client_negotiation.cc:769] Sending connection context
0811 03:05:26.650641 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 03:05:26.650850 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2361,"mutex_wait_us":3,"thread_start_us":2193,"threads_started":1}
I20260811 03:05:26.651921 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.637071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.27.109.1:41497)
0811 03:05:26.637823 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:26.637835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:26.638617 (+ 782us) server_negotiation.cc:407] Connection header received
0811 03:05:26.638989 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:26.638997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:26.639293 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:26.640354 (+ 1061us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:26.640635 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.642743 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.646877 (+ 4134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.648991 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.651042 (+ 2051us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.651332 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:26.651409 (+ 77us) server_negotiation.cc:299] Negotiation successful
0811 03:05:26.651611 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":362,"threads_started":1}
I20260811 03:05:26.652760 4886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.637517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.27.109.2:37217)
0811 03:05:26.639170 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:26.639181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:26.639459 (+ 278us) server_negotiation.cc:407] Connection header received
0811 03:05:26.639741 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:26.639747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:26.639971 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:26.640193 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:26.641583 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.643662 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.647339 (+ 3677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.649447 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.649631 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.649939 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:26.651979 (+ 2040us) server_negotiation.cc:299] Negotiation successful
0811 03:05:26.652212 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1455,"thread_start_us":1266,"threads_started":1}
I20260811 03:05:26.656368 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 (DEFAULT_TABLE table=test-table [id=e03ac93d416145869c2d34612d41a417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:26.657959 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:26.657728 4635 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 (DEFAULT_TABLE table=test-table [id=e03ac93d416145869c2d34612d41a417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:26.659211 4635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:26.661327 4887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.639712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:33703 (local address 127.0.0.1:42642)
0811 03:05:26.651599 (+ 11887us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:26.651620 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:26.651754 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:26.654060 (+ 2306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:26.654065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:26.654100 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:26.654288 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.654300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.656312 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.656318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.658216 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.658232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.660509 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.660515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.660711 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.660745 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:05:26.660851 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 03:05:26.661007 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11765,"thread_start_us":11948,"threads_started":1}
I20260811 03:05:26.661427 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.640299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.27.109.3:33703)
0811 03:05:26.653267 (+ 12968us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:26.653279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:26.653311 (+ 32us) server_negotiation.cc:407] Connection header received
0811 03:05:26.653458 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:26.653476 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:26.653834 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:26.654049 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:26.654488 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.656143 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.658425 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.659992 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.660175 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.660391 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:26.660934 (+ 543us) server_negotiation.cc:299] Negotiation successful
0811 03:05:26.661125 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12794,"thread_start_us":12658,"threads_started":1}
I20260811 03:05:26.665714 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 (DEFAULT_TABLE table=test-table [id=e03ac93d416145869c2d34612d41a417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:26.667305 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:26.672183 4893 tablet_bootstrap.cc:492] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Bootstrap starting.
I20260811 03:05:26.674449 4894 tablet_bootstrap.cc:492] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Bootstrap starting.
I20260811 03:05:26.677285 4893 tablet_bootstrap.cc:654] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:26.678349 4894 tablet_bootstrap.cc:654] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:26.678776 4893 log.cc:824] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:26.679692 4894 log.cc:824] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:26.682228 4893 tablet_bootstrap.cc:492] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: No bootstrap required, opened a new log
I20260811 03:05:26.682704 4893 ts_tablet_manager.cc:1397] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260811 03:05:26.683861 4894 tablet_bootstrap.cc:492] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: No bootstrap required, opened a new log
I20260811 03:05:26.684077 4897 tablet_bootstrap.cc:492] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Bootstrap starting.
I20260811 03:05:26.684185 4894 ts_tablet_manager.cc:1397] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260811 03:05:26.688012 4897 tablet_bootstrap.cc:654] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:26.689447 4897 log.cc:824] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:26.692663 4894 raft_consensus.cc:348] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.693168 4894 raft_consensus.cc:374] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:26.693284 4894 raft_consensus.cc:729] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13949c5b0e7845b988e01a3abf47c0d8, State: Initialized, Role: FOLLOWER
I20260811 03:05:26.694132 4894 consensus_queue.cc:260] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.694366 4897 tablet_bootstrap.cc:492] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: No bootstrap required, opened a new log
I20260811 03:05:26.694152 4893 raft_consensus.cc:348] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.694571 4893 raft_consensus.cc:374] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:26.694729 4897 ts_tablet_manager.cc:1397] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260811 03:05:26.694738 4893 raft_consensus.cc:729] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605bc85fec22464d8875471d34bad680, State: Initialized, Role: FOLLOWER
I20260811 03:05:26.695672 4893 consensus_queue.cc:260] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.696154 4894 ts_tablet_manager.cc:1428] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260811 03:05:26.697832 4893 ts_tablet_manager.cc:1428] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
W20260811 03:05:26.704226 4555 tablet.cc:2369] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:26.705082 4897 raft_consensus.cc:348] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.705500 4897 raft_consensus.cc:374] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:26.705606 4897 raft_consensus.cc:729] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3025f79396c740baa843ac83250eb65d, State: Initialized, Role: FOLLOWER
I20260811 03:05:26.706261 4897 consensus_queue.cc:260] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.707809 4897 ts_tablet_manager.cc:1428] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260811 03:05:26.790988 4710 tablet.cc:2369] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:26.827284 4900 raft_consensus.cc:482] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:26.827687 4900 raft_consensus.cc:504] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.829669 4900 leader_election.cc:288] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217), 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:26.837502 4886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.830160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:52999 (local address 127.27.109.2:37217)
0811 03:05:26.830396 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:26.830401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:26.831232 (+ 831us) server_negotiation.cc:407] Connection header received
0811 03:05:26.831463 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:26.831469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:26.831557 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:26.831674 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:26.832528 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.833501 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.835627 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.836453 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.836588 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.836643 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:26.837075 (+ 432us) server_negotiation.cc:299] Negotiation successful
0811 03:05:26.837225 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 03:05:26.837563 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.831498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:52985 (local address 127.27.109.3:33703)
0811 03:05:26.831749 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:26.831755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:26.832047 (+ 292us) server_negotiation.cc:407] Connection header received
0811 03:05:26.832174 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:26.832181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:26.832281 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:26.832386 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:26.833096 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.833923 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.835761 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.836587 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.836763 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.836840 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:26.837139 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 03:05:26.837335 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260811 03:05:26.837551 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.830138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37217 (local address 127.27.109.1:52999)
0811 03:05:26.831100 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:26.831122 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:26.831281 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:26.831855 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:26.831860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:26.831892 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:26.832316 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.832332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.833639 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.833645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.835437 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.835450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.836700 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.836707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.836899 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.836925 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 03:05:26.837003 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 03:05:26.837155 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":512,"threads_started":1}
I20260811 03:05:26.837620 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.831196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:33703 (local address 127.27.109.1:52985)
0811 03:05:26.831921 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:26.831936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:26.832066 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:26.832444 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:26.832448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:26.832474 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:26.832868 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.832884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.834071 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.834075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.835596 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:26.835614 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.836819 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.836824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:26.836941 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.836970 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 03:05:26.837089 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 03:05:26.837272 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":426,"threads_started":1}
I20260811 03:05:26.838918 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef4dcd4f92d492a978eb9c6bd9f73f3" candidate_uuid: "605bc85fec22464d8875471d34bad680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13949c5b0e7845b988e01a3abf47c0d8" is_pre_election: true
I20260811 03:05:26.838918 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef4dcd4f92d492a978eb9c6bd9f73f3" candidate_uuid: "605bc85fec22464d8875471d34bad680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3025f79396c740baa843ac83250eb65d" is_pre_election: true
I20260811 03:05:26.839581 4657 raft_consensus.cc:2454] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 605bc85fec22464d8875471d34bad680 in term 0.
I20260811 03:05:26.839581 4809 raft_consensus.cc:2454] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 605bc85fec22464d8875471d34bad680 in term 0.
I20260811 03:05:26.840384 4422 leader_election.cc:302] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3025f79396c740baa843ac83250eb65d, 605bc85fec22464d8875471d34bad680; no voters:
I20260811 03:05:26.840782 4900 raft_consensus.cc:2781] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:26.840955 4900 raft_consensus.cc:482] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:26.841106 4900 raft_consensus.cc:3037] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:26.843526 4900 raft_consensus.cc:504] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.844348 4900 leader_election.cc:288] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [CANDIDATE]: Term 1 election: Requested vote from peers 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217), 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:26.844812 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef4dcd4f92d492a978eb9c6bd9f73f3" candidate_uuid: "605bc85fec22464d8875471d34bad680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13949c5b0e7845b988e01a3abf47c0d8"
I20260811 03:05:26.845045 4657 raft_consensus.cc:3037] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:26.845321 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef4dcd4f92d492a978eb9c6bd9f73f3" candidate_uuid: "605bc85fec22464d8875471d34bad680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3025f79396c740baa843ac83250eb65d"
I20260811 03:05:26.845621 4809 raft_consensus.cc:3037] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:26.848582 4657 raft_consensus.cc:2454] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 605bc85fec22464d8875471d34bad680 in term 1.
I20260811 03:05:26.848977 4809 raft_consensus.cc:2454] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 605bc85fec22464d8875471d34bad680 in term 1.
I20260811 03:05:26.849092 4422 leader_election.cc:302] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13949c5b0e7845b988e01a3abf47c0d8, 605bc85fec22464d8875471d34bad680; no voters:
I20260811 03:05:26.849458 4900 raft_consensus.cc:2781] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:26.849985 4900 raft_consensus.cc:686] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 1 LEADER]: Becoming Leader. State: Replica: 605bc85fec22464d8875471d34bad680, State: Running, Role: LEADER
I20260811 03:05:26.850502 4900 consensus_queue.cc:237] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.855569 4326 catalog_manager.cc:5495] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 reported cstate change: term changed from 0 to 1, leader changed from <none> to 605bc85fec22464d8875471d34bad680 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "605bc85fec22464d8875471d34bad680" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
W20260811 03:05:26.884100 4864 tablet.cc:2369] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:05:26.904492 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:26.898372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.27.109.1:41497)
0811 03:05:26.898617 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:26.898621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:26.898641 (+ 20us) server_negotiation.cc:407] Connection header received
0811 03:05:26.898896 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:26.898900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:26.898984 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:26.899157 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:05:26.899650 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.900489 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.901972 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:26.902448 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:26.902549 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:26.903153 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:05:26.903759 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:05:26.903873 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:26.904057 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 03:05:26.904129 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260811 03:05:26.913769 4809 raft_consensus.cc:1261] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Refusing update from remote peer 605bc85fec22464d8875471d34bad680: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:26.913769 4657 raft_consensus.cc:1261] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Refusing update from remote peer 605bc85fec22464d8875471d34bad680: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:26.914935 4904 consensus_queue.cc:1035] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:26.915377 4900 consensus_queue.cc:1035] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:26.930186 4909 mvcc.cc:205] Tried to move back new op lower bound from 7317166190227812352 to 7317166189983899648. Current Snapshot: MvccSnapshot[applied={T|T < 7317166190227812352}]
I20260811 03:05:26.941254 4326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:26.941612 4326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:26.945595 4326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: Sending DeleteTablet for 3 replicas of tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3
I20260811 03:05:26.946969 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:26 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45636
I20260811 03:05:26.947096 4635 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:26 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:56388
I20260811 03:05:26.948076 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:26 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260811 03:05:26.948164 4326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:05:26.948853 4326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:05:26.951481 4919 tablet_replica.cc:330] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: stopping tablet replica
I20260811 03:05:26.952112 4920 tablet_replica.cc:330] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: stopping tablet replica
I20260811 03:05:26.952644 4920 raft_consensus.cc:2229] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:26.953163 4919 raft_consensus.cc:2229] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:26.953783 4920 raft_consensus.cc:2258] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:26.954278 4919 raft_consensus.cc:2258] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:26.954948 4917 tablet_replica.cc:330] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: stopping tablet replica
I20260811 03:05:26.956231 4917 raft_consensus.cc:2229] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:05:26.957083 4917 raft_consensus.cc:2258] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:26.958093 4920 ts_tablet_manager.cc:1905] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:26.959666 4919 ts_tablet_manager.cc:1905] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:26.959736 4917 ts_tablet_manager.cc:1905] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:26.961207 4785 tablet_service.cc:1467] Processing CreateTablet for tablet a889865502fa4d2aa54d3d642598ad5b (DEFAULT_TABLE table=test-table [id=e88c7904f3204f14848830000da4a263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:26.960487 4634 tablet_service.cc:1467] Processing CreateTablet for tablet a889865502fa4d2aa54d3d642598ad5b (DEFAULT_TABLE table=test-table [id=e88c7904f3204f14848830000da4a263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:26.961699 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a889865502fa4d2aa54d3d642598ad5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:26.961779 4634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a889865502fa4d2aa54d3d642598ad5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:26.967276 4478 tablet_service.cc:1467] Processing CreateTablet for tablet a889865502fa4d2aa54d3d642598ad5b (DEFAULT_TABLE table=test-table [id=e88c7904f3204f14848830000da4a263]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:26.967648 4917 ts_tablet_manager.cc:1918] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:26.967823 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a889865502fa4d2aa54d3d642598ad5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:26.969177 4894 tablet_bootstrap.cc:492] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: Bootstrap starting.
I20260811 03:05:26.971434 4894 tablet_bootstrap.cc:654] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:26.967824 4917 log.cc:1196] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/8ef4dcd4f92d492a978eb9c6bd9f73f3
I20260811 03:05:26.975041 4917 ts_tablet_manager.cc:1939] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 605bc85fec22464d8875471d34bad680: Deleting consensus metadata
I20260811 03:05:26.976709 4920 ts_tablet_manager.cc:1918] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:26.976759 4919 ts_tablet_manager.cc:1918] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:26.977005 4919 log.cc:1196] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/8ef4dcd4f92d492a978eb9c6bd9f73f3
I20260811 03:05:26.977149 4893 tablet_bootstrap.cc:492] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: Bootstrap starting.
I20260811 03:05:26.977192 4312 catalog_manager.cc:4841] TS 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497): tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 (table test-table [id=e03ac93d416145869c2d34612d41a417]) successfully deleted
I20260811 03:05:26.977640 4919 ts_tablet_manager.cc:1939] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting consensus metadata
I20260811 03:05:26.979234 4893 tablet_bootstrap.cc:654] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:26.979545 4311 catalog_manager.cc:4841] TS 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217): tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 (table test-table [id=e03ac93d416145869c2d34612d41a417]) successfully deleted
I20260811 03:05:26.980116 4897 tablet_bootstrap.cc:492] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: Bootstrap starting.
I20260811 03:05:26.980284 4920 log.cc:1196] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/8ef4dcd4f92d492a978eb9c6bd9f73f3
I20260811 03:05:26.980939 4920 ts_tablet_manager.cc:1939] T 8ef4dcd4f92d492a978eb9c6bd9f73f3 P 3025f79396c740baa843ac83250eb65d: Deleting consensus metadata
I20260811 03:05:26.982558 4897 tablet_bootstrap.cc:654] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:26.983952 4311 catalog_manager.cc:4841] TS 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703): tablet 8ef4dcd4f92d492a978eb9c6bd9f73f3 (table test-table [id=e03ac93d416145869c2d34612d41a417]) successfully deleted
I20260811 03:05:26.986523 4894 tablet_bootstrap.cc:492] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: No bootstrap required, opened a new log
I20260811 03:05:26.986829 4894 ts_tablet_manager.cc:1397] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20260811 03:05:26.987777 4894 raft_consensus.cc:348] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.988134 4894 raft_consensus.cc:374] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:26.988281 4894 raft_consensus.cc:729] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13949c5b0e7845b988e01a3abf47c0d8, State: Initialized, Role: FOLLOWER
I20260811 03:05:26.988629 4894 consensus_queue.cc:260] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.989354 4894 ts_tablet_manager.cc:1428] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 03:05:26.991510 4893 tablet_bootstrap.cc:492] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: No bootstrap required, opened a new log
I20260811 03:05:26.991757 4893 ts_tablet_manager.cc:1397] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260811 03:05:26.992762 4893 raft_consensus.cc:348] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.993103 4893 raft_consensus.cc:374] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:26.993273 4893 raft_consensus.cc:729] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605bc85fec22464d8875471d34bad680, State: Initialized, Role: FOLLOWER
I20260811 03:05:26.993682 4893 consensus_queue.cc:260] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.995317 4893 ts_tablet_manager.cc:1428] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260811 03:05:26.997573 4897 tablet_bootstrap.cc:492] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: No bootstrap required, opened a new log
I20260811 03:05:26.997766 4897 ts_tablet_manager.cc:1397] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20260811 03:05:26.998793 4897 raft_consensus.cc:348] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:26.999051 4897 raft_consensus.cc:374] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:26.999173 4897 raft_consensus.cc:729] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3025f79396c740baa843ac83250eb65d, State: Initialized, Role: FOLLOWER
I20260811 03:05:26.999461 4897 consensus_queue.cc:260] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.000232 4897 ts_tablet_manager.cc:1428] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 03:05:27.058353 4899 raft_consensus.cc:482] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:27.058645 4899 raft_consensus.cc:504] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.060665 4899 leader_election.cc:288] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497), 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:27.070030 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.061413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:45527 (local address 127.27.109.1:41497)
0811 03:05:27.061624 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.061631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.062219 (+ 588us) server_negotiation.cc:407] Connection header received
0811 03:05:27.062555 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.062562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.062709 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.062809 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:27.063556 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.064792 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.067549 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.068882 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.069073 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.069177 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.069512 (+ 335us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.069700 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 03:05:27.070078 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.060946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41497 (local address 127.27.109.2:45527)
0811 03:05:27.062046 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:27.062115 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:27.062351 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:27.062849 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:27.062856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:27.062888 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:27.063362 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.063374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.065038 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.065043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.067366 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.067381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.069131 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.069139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.069316 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.069353 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 03:05:27.069463 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 03:05:27.069635 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":77,"thread_start_us":1089,"threads_started":1}
I20260811 03:05:27.070349 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.061247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:38575 (local address 127.27.109.3:33703)
0811 03:05:27.061480 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.061490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.062532 (+ 1042us) server_negotiation.cc:407] Connection header received
0811 03:05:27.062729 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.062735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.062822 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.062919 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:27.063788 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.064863 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.067735 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.069050 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.069205 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.069279 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.069856 (+ 577us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.070021 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 03:05:27.071040 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.061047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:33703 (local address 127.27.109.2:38575)
0811 03:05:27.062426 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:27.062446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:27.062549 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:27.063069 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:27.063074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:27.063107 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:27.063622 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.063635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.064993 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.065001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.067554 (+ 2553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.067571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.069419 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.069448 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.069652 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.069689 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 03:05:27.070475 (+ 786us) client_negotiation.cc:240] Negotiation successful
0811 03:05:27.070634 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"mutex_wait_us":99,"thread_start_us":620,"threads_started":1}
I20260811 03:05:27.071687 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a889865502fa4d2aa54d3d642598ad5b" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605bc85fec22464d8875471d34bad680" is_pre_election: true
I20260811 03:05:27.072176 4506 raft_consensus.cc:2454] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 0.
I20260811 03:05:27.072876 4577 leader_election.cc:302] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13949c5b0e7845b988e01a3abf47c0d8, 605bc85fec22464d8875471d34bad680; no voters:
I20260811 03:05:27.073005 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a889865502fa4d2aa54d3d642598ad5b" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3025f79396c740baa843ac83250eb65d" is_pre_election: true
I20260811 03:05:27.073292 4809 raft_consensus.cc:2454] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 0.
I20260811 03:05:27.073340 4899 raft_consensus.cc:2781] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:27.073464 4899 raft_consensus.cc:482] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:27.073570 4899 raft_consensus.cc:3037] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.076066 4899 raft_consensus.cc:504] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.076664 4899 leader_election.cc:288] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 election: Requested vote from peers 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497), 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:27.077199 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a889865502fa4d2aa54d3d642598ad5b" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605bc85fec22464d8875471d34bad680"
I20260811 03:05:27.077237 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a889865502fa4d2aa54d3d642598ad5b" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3025f79396c740baa843ac83250eb65d"
I20260811 03:05:27.077440 4506 raft_consensus.cc:3037] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.077466 4809 raft_consensus.cc:3037] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.080389 4506 raft_consensus.cc:2454] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 1.
I20260811 03:05:27.080608 4809 raft_consensus.cc:2454] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 1.
I20260811 03:05:27.081007 4577 leader_election.cc:302] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13949c5b0e7845b988e01a3abf47c0d8, 605bc85fec22464d8875471d34bad680; no voters:
I20260811 03:05:27.081346 4899 raft_consensus.cc:2781] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:27.081858 4899 raft_consensus.cc:686] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 LEADER]: Becoming Leader. State: Replica: 13949c5b0e7845b988e01a3abf47c0d8, State: Running, Role: LEADER
I20260811 03:05:27.082322 4899 consensus_queue.cc:237] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.086496 4327 catalog_manager.cc:5495] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
Aug 11 03:05:27 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417523, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.62@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:27.142864 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.125188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39269 (local address 127.0.0.1:56928)
0811 03:05:27.126105 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:27.126132 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:27.126281 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:27.126795 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:27.126798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:27.127740 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:27.128119 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.128130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.129747 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.129752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.131132 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.131145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.132217 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.132221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.132357 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.133152 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:27.133171 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:27.135643 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:27.139948 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:27.139961 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:27.139968 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:27.140555 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:27.141010 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:27.141018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:27.141022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:27.141175 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:27.141879 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:27.141886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:27.142073 (+ 187us) client_negotiation.cc:769] Sending connection context
0811 03:05:27.142231 (+ 158us) client_negotiation.cc:240] Negotiation successful
0811 03:05:27.142471 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":412,"threads_started":1}
I20260811 03:05:27.143349 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.125267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.27.109.62:39269)
0811 03:05:27.126356 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.126363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.126389 (+ 26us) server_negotiation.cc:407] Connection header received
0811 03:05:27.126451 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.126456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.126579 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.126735 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:27.128325 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.129592 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.131309 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.131999 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.132136 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.135894 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:27.135901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:27.135906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:27.135944 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:27.139733 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:27.140692 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:27.140700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:27.140704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:27.140871 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:27.141319 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:27.141324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:27.141329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:27.141732 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:27.141888 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.142387 (+ 499us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.142632 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":738,"threads_started":1}
I20260811 03:05:27.149051 4327 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:56928:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 03:05:27.150319 4327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:05:27.158172 4635 tablet_service.cc:1467] Processing CreateTablet for tablet 8a74616638434cfabf446dcdcc8ca036 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83e3c93c12144181a109906f39c058db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:05:27.158205 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 8a74616638434cfabf446dcdcc8ca036 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83e3c93c12144181a109906f39c058db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:05:27.158864 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a74616638434cfabf446dcdcc8ca036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:27.158890 4635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a74616638434cfabf446dcdcc8ca036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:27.165069 4894 tablet_bootstrap.cc:492] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: Bootstrap starting.
I20260811 03:05:27.166320 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 8a74616638434cfabf446dcdcc8ca036 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83e3c93c12144181a109906f39c058db]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 03:05:27.166996 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a74616638434cfabf446dcdcc8ca036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:27.167007 4894 tablet_bootstrap.cc:654] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:27.167639 4893 tablet_bootstrap.cc:492] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: Bootstrap starting.
I20260811 03:05:27.169684 4893 tablet_bootstrap.cc:654] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:27.169757 4894 tablet_bootstrap.cc:492] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: No bootstrap required, opened a new log
I20260811 03:05:27.169919 4894 ts_tablet_manager.cc:1397] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 03:05:27.170882 4894 raft_consensus.cc:348] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.171265 4894 raft_consensus.cc:374] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:27.171367 4894 raft_consensus.cc:729] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13949c5b0e7845b988e01a3abf47c0d8, State: Initialized, Role: FOLLOWER
I20260811 03:05:27.171700 4894 consensus_queue.cc:260] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.172307 4899 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.172689 4899 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:05:27.173777 4894 ts_tablet_manager.cc:1428] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 03:05:27.177923 4897 tablet_bootstrap.cc:492] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: Bootstrap starting.
I20260811 03:05:27.179612 4893 tablet_bootstrap.cc:492] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: No bootstrap required, opened a new log
I20260811 03:05:27.179723 4897 tablet_bootstrap.cc:654] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:27.179885 4893 ts_tablet_manager.cc:1397] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260811 03:05:27.180737 4893 raft_consensus.cc:348] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.181133 4893 raft_consensus.cc:374] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:27.181268 4893 raft_consensus.cc:729] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605bc85fec22464d8875471d34bad680, State: Initialized, Role: FOLLOWER
I20260811 03:05:27.181587 4893 consensus_queue.cc:260] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.182140 4912 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.182552 4912 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:05:27.182657 4897 tablet_bootstrap.cc:492] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: No bootstrap required, opened a new log
I20260811 03:05:27.182866 4897 ts_tablet_manager.cc:1397] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260811 03:05:27.183187 4893 ts_tablet_manager.cc:1428] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260811 03:05:27.183684 4897 raft_consensus.cc:348] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.183954 4897 raft_consensus.cc:374] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:27.184062 4897 raft_consensus.cc:729] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3025f79396c740baa843ac83250eb65d, State: Initialized, Role: FOLLOWER
I20260811 03:05:27.184571 4897 consensus_queue.cc:260] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.185070 4901 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.185480 4901 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:05:27.186065 4897 ts_tablet_manager.cc:1428] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260811 03:05:27.473737 4899 raft_consensus.cc:482] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:27.473951 4899 raft_consensus.cc:504] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.474610 4899 leader_election.cc:288] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497), 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:27.475333 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a74616638434cfabf446dcdcc8ca036" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3025f79396c740baa843ac83250eb65d" is_pre_election: true
I20260811 03:05:27.475353 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a74616638434cfabf446dcdcc8ca036" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605bc85fec22464d8875471d34bad680" is_pre_election: true
I20260811 03:05:27.475652 4809 raft_consensus.cc:2454] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 0.
I20260811 03:05:27.475672 4506 raft_consensus.cc:2454] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 0.
I20260811 03:05:27.476219 4577 leader_election.cc:302] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13949c5b0e7845b988e01a3abf47c0d8, 605bc85fec22464d8875471d34bad680; no voters:
I20260811 03:05:27.476562 4899 raft_consensus.cc:2781] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:27.476661 4899 raft_consensus.cc:482] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:27.476744 4899 raft_consensus.cc:3037] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.479007 4899 raft_consensus.cc:504] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.479712 4899 leader_election.cc:288] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 election: Requested vote from peers 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497), 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:27.480350 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a74616638434cfabf446dcdcc8ca036" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605bc85fec22464d8875471d34bad680"
I20260811 03:05:27.480361 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a74616638434cfabf446dcdcc8ca036" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3025f79396c740baa843ac83250eb65d"
I20260811 03:05:27.480602 4506 raft_consensus.cc:3037] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.480634 4809 raft_consensus.cc:3037] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.483801 4809 raft_consensus.cc:2454] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 1.
I20260811 03:05:27.483800 4506 raft_consensus.cc:2454] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 1.
I20260811 03:05:27.484388 4576 leader_election.cc:302] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13949c5b0e7845b988e01a3abf47c0d8, 3025f79396c740baa843ac83250eb65d; no voters:
I20260811 03:05:27.484722 4899 raft_consensus.cc:2781] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:27.484939 4899 raft_consensus.cc:686] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 LEADER]: Becoming Leader. State: Replica: 13949c5b0e7845b988e01a3abf47c0d8, State: Running, Role: LEADER
I20260811 03:05:27.485271 4899 consensus_queue.cc:237] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.487318 4926 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: TxnStatusTablet state changed. Reason: New leader 13949c5b0e7845b988e01a3abf47c0d8. Latest consensus state: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.487632 4926 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:05:27.488747 4940 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 03:05:27.489068 4940 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 03:05:27.491258 4327 catalog_manager.cc:5495] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260811 03:05:27.535807 4926 consensus_queue.cc:1035] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:27.546022 4926 consensus_queue.cc:1035] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:27.570163 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.560994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37217 (local address 127.0.0.1:56392)
0811 03:05:27.561428 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:27.561449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:27.561582 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:27.563737 (+ 2155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:27.563749 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:27.563761 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:05:27.563936 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.563943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.565611 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.565615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.566907 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.566923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.568109 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.568113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.568264 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.568349 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:05:27.569586 (+ 1237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:05:27.569620 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:05:27.569694 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 03:05:27.569830 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260811 03:05:27.570248 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.560999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.27.109.2:37217)
0811 03:05:27.563170 (+ 2171us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.563178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.563240 (+ 62us) server_negotiation.cc:407] Connection header received
0811 03:05:27.563369 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.563375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.563499 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.563675 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:05:27.564093 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.565464 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.567164 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.567916 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.568060 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.568588 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:05:27.569446 (+ 858us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:05:27.569630 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.569739 (+ 109us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.569837 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2027,"thread_start_us":1866,"threads_started":1}
I20260811 03:05:27.574365 4809 raft_consensus.cc:1261] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Refusing update from remote peer 13949c5b0e7845b988e01a3abf47c0d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:27.574465 4506 raft_consensus.cc:1261] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Refusing update from remote peer 13949c5b0e7845b988e01a3abf47c0d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:27.575064 4926 consensus_queue.cc:1035] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:27.575408 4899 consensus_queue.cc:1035] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:27.579782 4912 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: TxnStatusTablet state changed. Reason: New leader 13949c5b0e7845b988e01a3abf47c0d8. Latest consensus state: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.580133 4912 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:05:27.580730 4901 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: TxnStatusTablet state changed. Reason: New leader 13949c5b0e7845b988e01a3abf47c0d8. Latest consensus state: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.581015 4901 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:05:27.583465 4943 mvcc.cc:205] Tried to move back new op lower bound from 7317166192937910272 to 7317166192582062080. Current Snapshot: MvccSnapshot[applied={T|T < 7317166192937910272}]
I20260811 03:05:27.586760 4912 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.587024 4912 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 605bc85fec22464d8875471d34bad680: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:05:27.591316 4899 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.591717 4899 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:05:27.594242 4901 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.594542 4901 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 3025f79396c740baa843ac83250eb65d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 03:05:27.608538 4926 tablet_replica.cc:439] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } } }
I20260811 03:05:27.608985 4926 tablet_replica.cc:442] T 8a74616638434cfabf446dcdcc8ca036 P 13949c5b0e7845b988e01a3abf47c0d8: This TxnStatusTablet replica's current role is: LEADER
I20260811 03:05:27.615955 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.608599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.27.109.2:37217)
0811 03:05:27.608777 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.608784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.609892 (+ 1108us) server_negotiation.cc:407] Connection header received
0811 03:05:27.610212 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.610217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.610330 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.610456 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:05:27.611055 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.611952 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.614060 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.614517 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.614619 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.615196 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:05:27.615302 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:05:27.615384 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.615550 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.615621 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 03:05:27.633661 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.627434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:42881 (local address 127.27.109.2:37217)
0811 03:05:27.627639 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.627644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.628272 (+ 628us) server_negotiation.cc:407] Connection header received
0811 03:05:27.628487 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.628492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.628582 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.628688 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:05:27.629195 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.630051 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.631755 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.632204 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.632356 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.632809 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:05:27.632916 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:05:27.633018 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.633156 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.633235 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260811 03:05:27.633693 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.627463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37217 (local address 127.27.109.2:42881)
0811 03:05:27.628158 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:27.628180 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:27.628357 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:27.628695 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:27.628700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:27.628711 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 03:05:27.629048 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.629059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.630199 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.630208 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.631602 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.631620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.632384 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.632390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.632559 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.632619 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 03:05:27.633018 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 03:05:27.633042 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 03:05:27.633137 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 03:05:27.633276 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":353,"threads_started":1}
I20260811 03:05:27.711109 4327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:27.711374 4327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:27.714496 4327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: Sending DeleteTablet for 3 replicas of tablet a889865502fa4d2aa54d3d642598ad5b
I20260811 03:05:27.715519 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet a889865502fa4d2aa54d3d642598ad5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45636
I20260811 03:05:27.716248 4965 tablet_replica.cc:330] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: stopping tablet replica
I20260811 03:05:27.716495 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet a889865502fa4d2aa54d3d642598ad5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260811 03:05:27.716609 4965 raft_consensus.cc:2229] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:27.716908 4965 raft_consensus.cc:2258] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:27.717160 4967 tablet_replica.cc:330] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: stopping tablet replica
I20260811 03:05:27.717463 4967 raft_consensus.cc:2229] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:27.717793 4967 raft_consensus.cc:2258] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:27.717762 4634 tablet_service.cc:1514] Processing DeleteTablet for tablet a889865502fa4d2aa54d3d642598ad5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:56388
I20260811 03:05:27.718313 4965 ts_tablet_manager.cc:1905] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:27.719000 4967 ts_tablet_manager.cc:1905] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:27.727133 4965 ts_tablet_manager.cc:1918] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:05:27.727383 4967 ts_tablet_manager.cc:1918] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:05:27.727427 4965 log.cc:1196] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/a889865502fa4d2aa54d3d642598ad5b
I20260811 03:05:27.727635 4967 log.cc:1196] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/a889865502fa4d2aa54d3d642598ad5b
I20260811 03:05:27.727953 4965 ts_tablet_manager.cc:1939] T a889865502fa4d2aa54d3d642598ad5b P 605bc85fec22464d8875471d34bad680: Deleting consensus metadata
I20260811 03:05:27.728204 4967 ts_tablet_manager.cc:1939] T a889865502fa4d2aa54d3d642598ad5b P 3025f79396c740baa843ac83250eb65d: Deleting consensus metadata
I20260811 03:05:27.729308 4312 catalog_manager.cc:4841] TS 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497): tablet a889865502fa4d2aa54d3d642598ad5b (table test-table [id=e88c7904f3204f14848830000da4a263]) successfully deleted
I20260811 03:05:27.735126 4311 catalog_manager.cc:4841] TS 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703): tablet a889865502fa4d2aa54d3d642598ad5b (table test-table [id=e88c7904f3204f14848830000da4a263]) successfully deleted
I20260811 03:05:27.735949 4968 tablet_replica.cc:330] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: stopping tablet replica
I20260811 03:05:27.736203 4968 raft_consensus.cc:2229] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:05:27.736799 4968 raft_consensus.cc:2258] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:27.737984 4968 ts_tablet_manager.cc:1905] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:27.745400 4968 ts_tablet_manager.cc:1918] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:05:27.745597 4968 log.cc:1196] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/a889865502fa4d2aa54d3d642598ad5b
I20260811 03:05:27.746057 4968 ts_tablet_manager.cc:1939] T a889865502fa4d2aa54d3d642598ad5b P 13949c5b0e7845b988e01a3abf47c0d8: Deleting consensus metadata
I20260811 03:05:27.747259 4311 catalog_manager.cc:4841] TS 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217): tablet a889865502fa4d2aa54d3d642598ad5b (table test-table [id=e88c7904f3204f14848830000da4a263]) successfully deleted
Aug 11 03:05:27 dist-test-slave-8tdl krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 03:05:27 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:27 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 03:05:27.822788 4976 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:27.799680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.27.109.62:39269)
0811 03:05:27.808598 (+ 8918us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.808605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.808634 (+ 29us) server_negotiation.cc:407] Connection header received
0811 03:05:27.808768 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.808774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.808878 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.809013 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:27.810936 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.812188 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.814135 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.814874 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.815004 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.822423 (+ 7419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56936: BlockingRecv error: recv got EOF from 127.0.0.1:56936 (error 108)
Metrics: {"server-negotiator.queue_time_us":8778,"thread_start_us":8639,"threads_started":1}
Aug 11 03:05:27 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM
I20260811 03:05:27.844611 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.826230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.27.109.62:39269)
0811 03:05:27.826430 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.826436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.827242 (+ 806us) server_negotiation.cc:407] Connection header received
0811 03:05:27.827425 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.827430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.827511 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.827613 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:27.829277 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.830204 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.832214 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.832706 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.832905 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.837820 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:27.837827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:27.837830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:27.837876 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:27.841362 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:27.842205 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:27.842210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:27.842213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:27.842367 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:27.842875 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:27.842879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:27.842888 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:27.843205 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:27.843508 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.843926 (+ 418us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.844216 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 03:05:27.848958 4327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:05:27.849689 4327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:05:27.857705 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 3418d5d108764a2ab3af9de0db7eeb66 (DEFAULT_TABLE table=test-table [id=61a54b24896547b68313e338a4ccda9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:27.857762 4634 tablet_service.cc:1467] Processing CreateTablet for tablet 3418d5d108764a2ab3af9de0db7eeb66 (DEFAULT_TABLE table=test-table [id=61a54b24896547b68313e338a4ccda9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:27.858239 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3418d5d108764a2ab3af9de0db7eeb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:27.858320 4634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3418d5d108764a2ab3af9de0db7eeb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:27.862180 4785 tablet_service.cc:1467] Processing CreateTablet for tablet 3418d5d108764a2ab3af9de0db7eeb66 (DEFAULT_TABLE table=test-table [id=61a54b24896547b68313e338a4ccda9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:27.862703 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3418d5d108764a2ab3af9de0db7eeb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:27.865115 4985 tablet_bootstrap.cc:492] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: Bootstrap starting.
I20260811 03:05:27.867211 4985 tablet_bootstrap.cc:654] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:27.875242 4987 tablet_bootstrap.cc:492] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: Bootstrap starting.
I20260811 03:05:27.879001 4987 tablet_bootstrap.cc:654] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:27.880146 4988 tablet_bootstrap.cc:492] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: Bootstrap starting.
I20260811 03:05:27.883534 4985 tablet_bootstrap.cc:492] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: No bootstrap required, opened a new log
I20260811 03:05:27.883580 4987 tablet_bootstrap.cc:492] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: No bootstrap required, opened a new log
I20260811 03:05:27.883745 4987 ts_tablet_manager.cc:1397] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260811 03:05:27.883752 4985 ts_tablet_manager.cc:1397] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260811 03:05:27.884064 4988 tablet_bootstrap.cc:654] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:27.884806 4987 raft_consensus.cc:348] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.885221 4987 raft_consensus.cc:374] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:27.885439 4987 raft_consensus.cc:729] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3025f79396c740baa843ac83250eb65d, State: Initialized, Role: FOLLOWER
I20260811 03:05:27.885936 4987 consensus_queue.cc:260] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.885969 4985 raft_consensus.cc:348] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.886274 4985 raft_consensus.cc:374] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:27.886392 4985 raft_consensus.cc:729] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13949c5b0e7845b988e01a3abf47c0d8, State: Initialized, Role: FOLLOWER
I20260811 03:05:27.886667 4985 consensus_queue.cc:260] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.887383 4985 ts_tablet_manager.cc:1428] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260811 03:05:27.890630 4987 ts_tablet_manager.cc:1428] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260811 03:05:27.895166 4988 tablet_bootstrap.cc:492] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: No bootstrap required, opened a new log
I20260811 03:05:27.895380 4988 ts_tablet_manager.cc:1397] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260811 03:05:27.896116 4988 raft_consensus.cc:348] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.896420 4988 raft_consensus.cc:374] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:27.896567 4988 raft_consensus.cc:729] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605bc85fec22464d8875471d34bad680, State: Initialized, Role: FOLLOWER
I20260811 03:05:27.897168 4988 consensus_queue.cc:260] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.898214 4901 raft_consensus.cc:482] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:27.898466 4901 raft_consensus.cc:504] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.899116 4988 ts_tablet_manager.cc:1428] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 03:05:27.900388 4901 leader_election.cc:288] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497), 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217)
I20260811 03:05:27.915104 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.900680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:41497 (local address 127.27.109.3:39945)
0811 03:05:27.907232 (+ 6552us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:27.907255 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:27.907400 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:27.908202 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:27.908207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:27.908245 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:27.908786 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.908801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.910455 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.910464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.912321 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.912335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.914030 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.914039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.914221 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.914260 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 03:05:27.914363 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 03:05:27.914514 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6393,"mutex_wait_us":82,"thread_start_us":7248,"threads_started":1}
I20260811 03:05:27.915966 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.901748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:39945 (local address 127.27.109.1:41497)
0811 03:05:27.907897 (+ 6149us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.907901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.907939 (+ 38us) server_negotiation.cc:407] Connection header received
0811 03:05:27.908007 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.908011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.908066 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.908173 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:27.909082 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.910293 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.912500 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.913603 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.913803 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.913899 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.915427 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.915621 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5980,"thread_start_us":5856,"threads_started":1}
I20260811 03:05:27.915992 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.902140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:38239 (local address 127.27.109.2:37217)
0811 03:05:27.902405 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.902415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.908378 (+ 5963us) server_negotiation.cc:407] Connection header received
0811 03:05:27.908738 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.908744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.908818 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.908926 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:05:27.909626 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.910420 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.913077 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.914548 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.914754 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.914830 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.915584 (+ 754us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.915724 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 03:05:27.916616 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418d5d108764a2ab3af9de0db7eeb66" candidate_uuid: "3025f79396c740baa843ac83250eb65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605bc85fec22464d8875471d34bad680" is_pre_election: true
I20260811 03:05:27.916776 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.901078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37217 (local address 127.27.109.3:38239)
0811 03:05:27.908252 (+ 7174us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:27.908268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:27.908458 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:27.908958 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:27.908963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:27.908989 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:05:27.909446 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.909457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.910569 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.910575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.912883 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:27.912900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.915004 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.915012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:27.915360 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.915403 (+ 43us) client_negotiation.cc:769] Sending connection context
0811 03:05:27.916244 (+ 841us) client_negotiation.cc:240] Negotiation successful
0811 03:05:27.916386 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7040,"mutex_wait_us":25,"thread_start_us":5808,"threads_started":1}
I20260811 03:05:27.916927 4506 raft_consensus.cc:2454] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3025f79396c740baa843ac83250eb65d in term 0.
I20260811 03:05:27.917423 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418d5d108764a2ab3af9de0db7eeb66" candidate_uuid: "3025f79396c740baa843ac83250eb65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13949c5b0e7845b988e01a3abf47c0d8" is_pre_election: true
I20260811 03:05:27.917577 4734 leader_election.cc:302] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3025f79396c740baa843ac83250eb65d, 605bc85fec22464d8875471d34bad680; no voters:
I20260811 03:05:27.917706 4657 raft_consensus.cc:2454] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3025f79396c740baa843ac83250eb65d in term 0.
I20260811 03:05:27.918126 4901 raft_consensus.cc:2781] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:27.918275 4901 raft_consensus.cc:482] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:27.918383 4901 raft_consensus.cc:3037] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.920678 4901 raft_consensus.cc:504] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.921347 4901 leader_election.cc:288] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [CANDIDATE]: Term 1 election: Requested vote from peers 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497), 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217)
I20260811 03:05:27.921828 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418d5d108764a2ab3af9de0db7eeb66" candidate_uuid: "3025f79396c740baa843ac83250eb65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605bc85fec22464d8875471d34bad680"
I20260811 03:05:27.921990 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3418d5d108764a2ab3af9de0db7eeb66" candidate_uuid: "3025f79396c740baa843ac83250eb65d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13949c5b0e7845b988e01a3abf47c0d8"
I20260811 03:05:27.922093 4506 raft_consensus.cc:3037] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.922230 4657 raft_consensus.cc:3037] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:27.924634 4506 raft_consensus.cc:2454] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3025f79396c740baa843ac83250eb65d in term 1.
I20260811 03:05:27.925179 4734 leader_election.cc:302] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3025f79396c740baa843ac83250eb65d, 605bc85fec22464d8875471d34bad680; no voters:
I20260811 03:05:27.925513 4901 raft_consensus.cc:2781] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:27.925670 4657 raft_consensus.cc:2454] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3025f79396c740baa843ac83250eb65d in term 1.
I20260811 03:05:27.926100 4901 raft_consensus.cc:686] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 1 LEADER]: Becoming Leader. State: Replica: 3025f79396c740baa843ac83250eb65d, State: Running, Role: LEADER
I20260811 03:05:27.926600 4901 consensus_queue.cc:237] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:27.930365 4326 catalog_manager.cc:5495] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3025f79396c740baa843ac83250eb65d (127.27.109.3). New cstate: current_term: 1 leader_uuid: "3025f79396c740baa843ac83250eb65d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260811 03:05:27.953161 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:27.946755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.27.109.3:33703)
0811 03:05:27.947391 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:27.947396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:27.947415 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:05:27.947508 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:27.947512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:27.947600 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:27.947784 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:05:27.948245 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.949072 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.950467 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:27.951002 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:27.951160 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:27.951657 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:05:27.952311 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:05:27.952483 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:27.952614 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 03:05:27.952706 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":342,"threads_started":1}
I20260811 03:05:27.957543 4657 raft_consensus.cc:1261] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Refusing update from remote peer 3025f79396c740baa843ac83250eb65d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:27.958308 4506 raft_consensus.cc:1261] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Refusing update from remote peer 3025f79396c740baa843ac83250eb65d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:27.958544 4994 consensus_queue.cc:1035] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [LEADER]: Connected to new peer: Peer: permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:27.959054 4901 consensus_queue.cc:1035] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [LEADER]: Connected to new peer: Peer: permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:27.981528 4327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:27.981783 4327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:27.985497 4327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: Sending DeleteTablet for 3 replicas of tablet 3418d5d108764a2ab3af9de0db7eeb66
I20260811 03:05:27.986591 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 3418d5d108764a2ab3af9de0db7eeb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45636
I20260811 03:05:27.986752 4477 tablet_service.cc:1514] Processing DeleteTablet for tablet 3418d5d108764a2ab3af9de0db7eeb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45636
I20260811 03:05:27.987046 4634 tablet_service.cc:1514] Processing DeleteTablet for tablet 3418d5d108764a2ab3af9de0db7eeb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:56388
I20260811 03:05:27.987375 4968 tablet_replica.cc:330] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: stopping tablet replica
I20260811 03:05:27.987396 4965 tablet_replica.cc:330] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: stopping tablet replica
I20260811 03:05:27.987599 4965 raft_consensus.cc:2229] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:27.987607 4968 raft_consensus.cc:2229] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:27.987849 4968 raft_consensus.cc:2258] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:27.987861 4965 raft_consensus.cc:2258] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:27.989202 4965 ts_tablet_manager.cc:1905] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:27.990113 4326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:05:27.990746 4326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:05:27.990931 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet 3418d5d108764a2ab3af9de0db7eeb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:27 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260811 03:05:27.991248 4967 tablet_replica.cc:330] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: stopping tablet replica
I20260811 03:05:27.991456 4967 raft_consensus.cc:2229] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:05:27.992020 4967 raft_consensus.cc:2258] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:27.993129 4967 ts_tablet_manager.cc:1905] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:27.993476 4968 ts_tablet_manager.cc:1905] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:27.998302 4965 ts_tablet_manager.cc:1918] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:27.998499 4965 log.cc:1196] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/3418d5d108764a2ab3af9de0db7eeb66
I20260811 03:05:27.998965 4965 ts_tablet_manager.cc:1939] T 3418d5d108764a2ab3af9de0db7eeb66 P 605bc85fec22464d8875471d34bad680: Deleting consensus metadata
I20260811 03:05:28.000746 4967 ts_tablet_manager.cc:1918] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:28.001045 4967 log.cc:1196] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/3418d5d108764a2ab3af9de0db7eeb66
I20260811 03:05:28.001672 4967 ts_tablet_manager.cc:1939] T 3418d5d108764a2ab3af9de0db7eeb66 P 3025f79396c740baa843ac83250eb65d: Deleting consensus metadata
I20260811 03:05:28.002723 4968 ts_tablet_manager.cc:1918] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:05:28.002912 4968 log.cc:1196] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/3418d5d108764a2ab3af9de0db7eeb66
I20260811 03:05:28.003093 4311 catalog_manager.cc:4841] TS 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703): tablet 3418d5d108764a2ab3af9de0db7eeb66 (table test-table [id=61a54b24896547b68313e338a4ccda9f]) successfully deleted
I20260811 03:05:28.003402 4968 ts_tablet_manager.cc:1939] T 3418d5d108764a2ab3af9de0db7eeb66 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting consensus metadata
I20260811 03:05:28.005529 4477 tablet_service.cc:1467] Processing CreateTablet for tablet c1753b86ffd64b7cbec1224353a4efc0 (DEFAULT_TABLE table=test-table [id=a544b0ae375f4368b072cc56503592f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:28.005945 4312 catalog_manager.cc:4841] TS 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497): tablet 3418d5d108764a2ab3af9de0db7eeb66 (table test-table [id=61a54b24896547b68313e338a4ccda9f]) successfully deleted
I20260811 03:05:28.006068 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1753b86ffd64b7cbec1224353a4efc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:28.008723 4311 catalog_manager.cc:4841] TS 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217): tablet 3418d5d108764a2ab3af9de0db7eeb66 (table test-table [id=61a54b24896547b68313e338a4ccda9f]) successfully deleted
I20260811 03:05:28.009645 4785 tablet_service.cc:1467] Processing CreateTablet for tablet c1753b86ffd64b7cbec1224353a4efc0 (DEFAULT_TABLE table=test-table [id=a544b0ae375f4368b072cc56503592f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:28.010166 4785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1753b86ffd64b7cbec1224353a4efc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260811 03:05:28.008725 4312 catalog_manager.cc:4808] TS 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497): delete failed for tablet 3418d5d108764a2ab3af9de0db7eeb66 because the tablet was not found. No further retry: Not found: Tablet not found: 3418d5d108764a2ab3af9de0db7eeb66
I20260811 03:05:28.015612 4634 tablet_service.cc:1467] Processing CreateTablet for tablet c1753b86ffd64b7cbec1224353a4efc0 (DEFAULT_TABLE table=test-table [id=a544b0ae375f4368b072cc56503592f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:05:28.016129 4634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1753b86ffd64b7cbec1224353a4efc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:28.021667 4988 tablet_bootstrap.cc:492] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: Bootstrap starting.
I20260811 03:05:28.023631 4988 tablet_bootstrap.cc:654] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:28.029232 4987 tablet_bootstrap.cc:492] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: Bootstrap starting.
I20260811 03:05:28.029430 4985 tablet_bootstrap.cc:492] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: Bootstrap starting.
I20260811 03:05:28.031225 4985 tablet_bootstrap.cc:654] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:28.031594 4987 tablet_bootstrap.cc:654] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:28.039728 4988 tablet_bootstrap.cc:492] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: No bootstrap required, opened a new log
I20260811 03:05:28.040062 4988 ts_tablet_manager.cc:1397] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.016s
I20260811 03:05:28.041143 4988 raft_consensus.cc:348] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.041595 4988 raft_consensus.cc:374] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:28.041750 4988 raft_consensus.cc:729] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605bc85fec22464d8875471d34bad680, State: Initialized, Role: FOLLOWER
I20260811 03:05:28.042164 4988 consensus_queue.cc:260] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.044486 4988 ts_tablet_manager.cc:1428] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260811 03:05:28.045423 4985 tablet_bootstrap.cc:492] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: No bootstrap required, opened a new log
I20260811 03:05:28.045455 4987 tablet_bootstrap.cc:492] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: No bootstrap required, opened a new log
I20260811 03:05:28.045612 4985 ts_tablet_manager.cc:1397] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260811 03:05:28.045641 4987 ts_tablet_manager.cc:1397] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.009s
I20260811 03:05:28.046533 4987 raft_consensus.cc:348] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.046835 4987 raft_consensus.cc:374] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:28.046918 4987 raft_consensus.cc:729] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3025f79396c740baa843ac83250eb65d, State: Initialized, Role: FOLLOWER
I20260811 03:05:28.047204 4987 consensus_queue.cc:260] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.047818 4987 ts_tablet_manager.cc:1428] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 03:05:28.051257 4985 raft_consensus.cc:348] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.051577 4985 raft_consensus.cc:374] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:28.051671 4985 raft_consensus.cc:729] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13949c5b0e7845b988e01a3abf47c0d8, State: Initialized, Role: FOLLOWER
I20260811 03:05:28.051944 4985 consensus_queue.cc:260] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.052589 4985 ts_tablet_manager.cc:1428] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260811 03:05:28.194990 4963 raft_consensus.cc:482] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:28.195343 4963 raft_consensus.cc:504] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.196277 4963 leader_election.cc:288] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497), 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:28.196892 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1753b86ffd64b7cbec1224353a4efc0" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605bc85fec22464d8875471d34bad680" is_pre_election: true
I20260811 03:05:28.197055 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1753b86ffd64b7cbec1224353a4efc0" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3025f79396c740baa843ac83250eb65d" is_pre_election: true
I20260811 03:05:28.197206 4506 raft_consensus.cc:2454] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 0.
I20260811 03:05:28.197376 4809 raft_consensus.cc:2454] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 0.
I20260811 03:05:28.197913 4576 leader_election.cc:302] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13949c5b0e7845b988e01a3abf47c0d8, 3025f79396c740baa843ac83250eb65d; no voters:
I20260811 03:05:28.198284 4963 raft_consensus.cc:2781] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:28.198418 4963 raft_consensus.cc:482] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:28.198521 4963 raft_consensus.cc:3037] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:28.203058 4963 raft_consensus.cc:504] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.204396 4506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1753b86ffd64b7cbec1224353a4efc0" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605bc85fec22464d8875471d34bad680"
I20260811 03:05:28.204656 4506 raft_consensus.cc:3037] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:28.208132 4506 raft_consensus.cc:2454] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 1.
I20260811 03:05:28.208741 4577 leader_election.cc:302] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13949c5b0e7845b988e01a3abf47c0d8, 605bc85fec22464d8875471d34bad680; no voters:
I20260811 03:05:28.214751 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1753b86ffd64b7cbec1224353a4efc0" candidate_uuid: "13949c5b0e7845b988e01a3abf47c0d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3025f79396c740baa843ac83250eb65d"
I20260811 03:05:28.215075 4809 raft_consensus.cc:3037] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:28.218390 4809 raft_consensus.cc:2454] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13949c5b0e7845b988e01a3abf47c0d8 in term 1.
I20260811 03:05:28.220958 4963 leader_election.cc:288] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [CANDIDATE]: Term 1 election: Requested vote from peers 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497), 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703)
I20260811 03:05:28.221481 4963 raft_consensus.cc:2781] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:28.221923 4963 raft_consensus.cc:686] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 LEADER]: Becoming Leader. State: Replica: 13949c5b0e7845b988e01a3abf47c0d8, State: Running, Role: LEADER
I20260811 03:05:28.222425 4963 consensus_queue.cc:237] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } }
I20260811 03:05:28.232918 4327 catalog_manager.cc:5495] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "13949c5b0e7845b988e01a3abf47c0d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13949c5b0e7845b988e01a3abf47c0d8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260811 03:05:28.291985 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:28.286758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.27.109.2:37217)
0811 03:05:28.286941 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:28.286947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:28.287085 (+ 138us) server_negotiation.cc:407] Connection header received
0811 03:05:28.287303 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:28.287307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:28.287397 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:28.287500 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:05:28.287906 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:28.288646 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:28.290012 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:28.290517 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:28.290675 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:28.291145 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:05:28.291284 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:05:28.291387 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:28.291550 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 03:05:28.291651 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:05:28.305279 4809 raft_consensus.cc:1261] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Refusing update from remote peer 13949c5b0e7845b988e01a3abf47c0d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:28.305960 4506 raft_consensus.cc:1261] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Refusing update from remote peer 13949c5b0e7845b988e01a3abf47c0d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:05:28.306330 4963 consensus_queue.cc:1035] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3025f79396c740baa843ac83250eb65d" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:28.306622 5005 consensus_queue.cc:1035] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605bc85fec22464d8875471d34bad680" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:05:28.335711 4945 mvcc.cc:205] Tried to move back new op lower bound from 7317166195932991488 to 7317166195607805952. Current Snapshot: MvccSnapshot[applied={T|T < 7317166195932991488}]
I20260811 03:05:28.402192 4327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:28.402494 4327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:05:28.406204 4327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03f4e85b7f2149cfa82b3fc58bafbf8b: Sending DeleteTablet for 3 replicas of tablet c1753b86ffd64b7cbec1224353a4efc0
I20260811 03:05:28.407052 4477 tablet_service.cc:1514] Processing DeleteTablet for tablet c1753b86ffd64b7cbec1224353a4efc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:28 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45636
I20260811 03:05:28.407420 4965 tablet_replica.cc:330] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: stopping tablet replica
I20260811 03:05:28.407624 4965 raft_consensus.cc:2229] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:28.407850 4965 raft_consensus.cc:2258] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:28.408766 4634 tablet_service.cc:1514] Processing DeleteTablet for tablet c1753b86ffd64b7cbec1224353a4efc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:28 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:56388
I20260811 03:05:28.409101 4968 tablet_replica.cc:330] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: stopping tablet replica
I20260811 03:05:28.409345 4968 raft_consensus.cc:2229] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:05:28.409729 4965 ts_tablet_manager.cc:1905] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:28.409735 4968 raft_consensus.cc:2258] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:28.410989 4968 ts_tablet_manager.cc:1905] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:28.411109 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 4403
I20260811 03:05:28.414891 4785 tablet_service.cc:1514] Processing DeleteTablet for tablet c1753b86ffd64b7cbec1224353a4efc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:05:28 UTC) from {username='oryx', principal='oryx/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:42642
I20260811 03:05:28.415267 4967 tablet_replica.cc:330] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: stopping tablet replica
I20260811 03:05:28.415477 4967 raft_consensus.cc:2229] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:05:28.415711 4967 raft_consensus.cc:2258] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:05:28.417872 4968 ts_tablet_manager.cc:1918] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:05:28.417987 4965 ts_tablet_manager.cc:1918] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:05:28.418028 4968 log.cc:1196] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/c1753b86ffd64b7cbec1224353a4efc0
I20260811 03:05:28.418143 4965 log.cc:1196] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/c1753b86ffd64b7cbec1224353a4efc0
I20260811 03:05:28.418514 4968 ts_tablet_manager.cc:1939] T c1753b86ffd64b7cbec1224353a4efc0 P 13949c5b0e7845b988e01a3abf47c0d8: Deleting consensus metadata
I20260811 03:05:28.418638 4965 ts_tablet_manager.cc:1939] T c1753b86ffd64b7cbec1224353a4efc0 P 605bc85fec22464d8875471d34bad680: Deleting consensus metadata
I20260811 03:05:28.419916 4312 catalog_manager.cc:4841] TS 605bc85fec22464d8875471d34bad680 (127.27.109.1:41497): tablet c1753b86ffd64b7cbec1224353a4efc0 (table test-table [id=a544b0ae375f4368b072cc56503592f7]) successfully deleted
I20260811 03:05:28.422614 4967 ts_tablet_manager.cc:1905] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:05:28.428016 4311 catalog_manager.cc:4841] TS 13949c5b0e7845b988e01a3abf47c0d8 (127.27.109.2:37217): tablet c1753b86ffd64b7cbec1224353a4efc0 (table test-table [id=a544b0ae375f4368b072cc56503592f7]) successfully deleted
I20260811 03:05:28.430783 4967 ts_tablet_manager.cc:1918] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 03:05:28.430974 4967 log.cc:1196] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/c1753b86ffd64b7cbec1224353a4efc0
I20260811 03:05:28.431442 4967 ts_tablet_manager.cc:1939] T c1753b86ffd64b7cbec1224353a4efc0 P 3025f79396c740baa843ac83250eb65d: Deleting consensus metadata
I20260811 03:05:28.433002 4311 catalog_manager.cc:4841] TS 3025f79396c740baa843ac83250eb65d (127.27.109.3:33703): tablet c1753b86ffd64b7cbec1224353a4efc0 (table test-table [id=a544b0ae375f4368b072cc56503592f7]) successfully deleted
Aug 11 03:05:28 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.1@KRBTEST.COM
I20260811 03:05:28.450228 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:28.413437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.27.109.1:41497)
0811 03:05:28.413611 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:28.413618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:28.428274 (+ 14656us) server_negotiation.cc:407] Connection header received
0811 03:05:28.428633 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:28.428641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:28.428739 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:28.428820 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:28.431027 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:28.432158 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:28.434148 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:28.434925 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:28.435052 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:28.443723 (+ 8671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:28.443743 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:28.443761 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:28.443813 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:28.447039 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:28.447846 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:28.447853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:28.447856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:28.448007 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:28.448452 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:28.448457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:28.448459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:28.448845 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:28.449176 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:28.449489 (+ 313us) server_negotiation.cc:299] Negotiation successful
0811 03:05:28.449766 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 03:05:28.451061 4549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:28.608695 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 4403
I20260811 03:05:28.638581 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 4558
Aug 11 03:05:28 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.2@KRBTEST.COM
I20260811 03:05:28.656335 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:28.640116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.27.109.2:37217)
0811 03:05:28.640337 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:28.640341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:28.640358 (+ 17us) server_negotiation.cc:407] Connection header received
0811 03:05:28.640423 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:28.640427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:28.640494 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:28.640615 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:28.642110 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:28.643140 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:28.644494 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:28.644958 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:28.645098 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:28.649657 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:28.649674 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:28.649691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:28.649738 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:28.653049 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:28.653847 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:28.653855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:28.653858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:28.654009 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:28.654412 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:28.654417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:28.654423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:28.655004 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:28.655434 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:28.655746 (+ 312us) server_negotiation.cc:299] Negotiation successful
0811 03:05:28.656003 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 03:05:28.657117 4704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:28.819779 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 4558
W20260811 03:05:28.848671 4732 connection.cc:535] server connection from 127.27.109.2:38575 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 03:05:28.849347 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 4713
Aug 11 03:05:28 dist-test-slave-8tdl krb5kdc[4275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.109.3@KRBTEST.COM
I20260811 03:05:28.867092 5015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:28.850080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.27.109.3:33703)
0811 03:05:28.850597 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:28.850602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:28.850620 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:05:28.850737 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:28.850741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:28.850809 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:28.850904 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:28.852302 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:28.853191 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:28.854647 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:28.855178 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:28.855301 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:28.860639 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:28.860658 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:28.860675 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:28.860723 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:28.863931 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:28.864721 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:28.864730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:28.864732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:28.864909 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:28.865372 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:28.865376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:28.865378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:28.865796 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:28.866109 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:28.866404 (+ 295us) server_negotiation.cc:299] Negotiation successful
0811 03:05:28.866601 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":235,"threads_started":1}
I20260811 03:05:28.867942 4858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:28.998574 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 4713
I20260811 03:05:29.027920 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 4291
I20260811 03:05:29.029387 4371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:29.154069 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 4291
2026-08-11T03:05:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7663 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:05:29 dist-test-slave-8tdl krb5kdc[5020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:05:29 dist-test-slave-8tdl krb5kdc[5020](info): set up 2 sockets
Aug 11 03:05:29 dist-test-slave-8tdl krb5kdc[5020](info): commencing operation
krb5kdc: starting...
W20260811 03:05:30.369807 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:05:30 dist-test-slave-8tdl krb5kdc[5020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:05:30.682500 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33225
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:33225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:31.055187 5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:31.055539 5035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:31.055635 5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:31.063242 5035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:31.063364 5035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:31.063419 5035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:31.063478 5035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:31.063529 5035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:31.074601 5035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:33225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:33225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:31.075995 5035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:31.077446 5035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:31.091223 5040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:31.091147 5041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:31.091274 5035 server_base.cc:1034] running on GCE node
W20260811 03:05:31.091187 5043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:31.092667 5035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:31.094283 5035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:31.095511 5035 hybrid_clock.cc:648] HybridClock initialized: now 1786417531095463 us; error 40 us; skew 500 ppm
I20260811 03:05:31.095911 5035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-11T03:05:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2010 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:05:31 dist-test-slave-8tdl krb5kdc[5049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:05:31 dist-test-slave-8tdl krb5kdc[5049](info): set up 2 sockets
Aug 11 03:05:31 dist-test-slave-8tdl krb5kdc[5049](info): commencing operation
krb5kdc: starting...
W20260811 03:05:32.385589 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:05:32 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:05:32.718112 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34745
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:33.101789 5062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:33.102140 5062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:33.102239 5062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:33.110445 5062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:33.110569 5062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:33.110616 5062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:33.110745 5062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:33.110800 5062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:33.121280 5062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:33.123098 5062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:33.124511 5062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:33.138028 5067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:33.138024 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:33.137974 5069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:33.139261 5062 server_base.cc:1034] running on GCE node
I20260811 03:05:33.140153 5062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:33.141953 5062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:33.143198 5062 hybrid_clock.cc:648] HybridClock initialized: now 1786417533143140 us; error 48 us; skew 500 ppm
I20260811 03:05:33.143572 5062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:33 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417533, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:33.147179 5062 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:05:33.149024 5062 webserver.cc:460] Webserver started at http://127.27.109.62:34769/ using document root <none> and password file <none>
I20260811 03:05:33.149808 5062 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:33.149958 5062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:33.150306 5062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:33.153587 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "223d965189844e348507d547bcb57775"
format_stamp: "Formatted at 2026-08-11 03:05:33 on dist-test-slave-8tdl"
server_key: "51683eca1d72aa8d15eb62d3f928a9da"
server_key_iv: "c25f731ec67214132493af370c0e1564"
server_key_version: "encryptionkey@0"
I20260811 03:05:33.154428 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "223d965189844e348507d547bcb57775"
format_stamp: "Formatted at 2026-08-11 03:05:33 on dist-test-slave-8tdl"
server_key: "51683eca1d72aa8d15eb62d3f928a9da"
server_key_iv: "c25f731ec67214132493af370c0e1564"
server_key_version: "encryptionkey@0"
I20260811 03:05:33.160171 5062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260811 03:05:33.164026 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:33.165527 5062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:05:33.165828 5062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "223d965189844e348507d547bcb57775"
format_stamp: "Formatted at 2026-08-11 03:05:33 on dist-test-slave-8tdl"
server_key: "51683eca1d72aa8d15eb62d3f928a9da"
server_key_iv: "c25f731ec67214132493af370c0e1564"
server_key_version: "encryptionkey@0"
I20260811 03:05:33.166076 5062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:33.192366 5062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:33.197136 5062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:33.197495 5062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:33.218556 5062 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:34745
I20260811 03:05:33.218595 5135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:34745 every 8 connection(s)
I20260811 03:05:33.220088 5062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:33.220726 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 5062
I20260811 03:05:33.221029 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:33.221838 28084 external_mini_cluster.cc:1428] Setting key 7b4214e0375880a73fc148f9d30283f0
I20260811 03:05:33.224637 5137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 03:05:33 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:05:33.238897 5137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Bootstrap starting.
I20260811 03:05:33.243528 5137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:33.244946 5137 log.cc:824] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:33.247641 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:33.224094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.27.109.62:34745)
0811 03:05:33.224950 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:33.224962 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:33.225003 (+ 41us) server_negotiation.cc:407] Connection header received
0811 03:05:33.226897 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:33.226919 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:33.227313 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:33.227671 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:33.229591 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:33.231094 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:33.232900 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:33.233590 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:33.233728 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:33.238633 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:33.238646 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:33.238667 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:33.238757 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:33.242851 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:33.243832 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:33.243839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:33.243845 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:33.244071 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:33.244532 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:33.244535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:33.244537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:33.245142 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:33.245400 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:33.246038 (+ 638us) server_negotiation.cc:299] Negotiation successful
0811 03:05:33.246539 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":363,"threads_started":1}
I20260811 03:05:33.248558 5137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: No bootstrap required, opened a new log
I20260811 03:05:33.257572 5137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } }
I20260811 03:05:33.258052 5137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:33.258178 5137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223d965189844e348507d547bcb57775, State: Initialized, Role: FOLLOWER
I20260811 03:05:33.258929 5137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } }
I20260811 03:05:33.259202 5137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:33.259378 5137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:33.259578 5137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:33.262099 5137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } }
I20260811 03:05:33.262665 5137 leader_election.cc:302] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 223d965189844e348507d547bcb57775; no voters:
I20260811 03:05:33.263443 5137 leader_election.cc:288] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:33.263645 5143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:33.264508 5143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 1 LEADER]: Becoming Leader. State: Replica: 223d965189844e348507d547bcb57775, State: Running, Role: LEADER
I20260811 03:05:33.265197 5143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } }
I20260811 03:05:33.265894 5137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:33.269201 5146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 223d965189844e348507d547bcb57775. Latest consensus state: current_term: 1 leader_uuid: "223d965189844e348507d547bcb57775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } } }
I20260811 03:05:33.269702 5146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:33.270205 5144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "223d965189844e348507d547bcb57775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } } }
I20260811 03:05:33.270694 5144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:33.276171 5151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:33.281023 5151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:33.290620 5151 catalog_manager.cc:1347] Generated new cluster ID: a6c6a6882b6549e7b878697d54ddc466
I20260811 03:05:33.290798 5151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:33.319355 5151 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:33.320858 5151 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:33.326946 5151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Generated new TSK 0
I20260811 03:05:33.327895 5151 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:05:33.416127 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34745
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:33.807691 5168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:33.808002 5168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:33.808101 5168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:33.815613 5168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:33.815743 5168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:33.815860 5168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:05:33.826053 5168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:33.827601 5168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:33.828977 5168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:33.843245 5174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:33.843199 5177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:33.843293 5175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:33.844686 5168 server_base.cc:1034] running on GCE node
I20260811 03:05:33.845410 5168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:33.846570 5168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:33.847790 5168 hybrid_clock.cc:648] HybridClock initialized: now 1786417533847742 us; error 42 us; skew 500 ppm
I20260811 03:05:33.848215 5168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:33 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417533, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:33.855477 5168 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:05:33.857254 5168 webserver.cc:460] Webserver started at http://127.27.109.1:43389/ using document root <none> and password file <none>
I20260811 03:05:33.857966 5168 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:33.858085 5168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:33.858356 5168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:33.861647 5168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "8f1a0277ffdc46fa9d17fa20917c2ae3"
format_stamp: "Formatted at 2026-08-11 03:05:33 on dist-test-slave-8tdl"
server_key: "8a0f363d3ffbfe9d80fb8eee3f4a219a"
server_key_iv: "a6814b4b0cacdae7273c7b99414f46f3"
server_key_version: "encryptionkey@0"
I20260811 03:05:33.862484 5168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "8f1a0277ffdc46fa9d17fa20917c2ae3"
format_stamp: "Formatted at 2026-08-11 03:05:33 on dist-test-slave-8tdl"
server_key: "8a0f363d3ffbfe9d80fb8eee3f4a219a"
server_key_iv: "a6814b4b0cacdae7273c7b99414f46f3"
server_key_version: "encryptionkey@0"
I20260811 03:05:33.868247 5168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 03:05:33.872521 5184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:33.874090 5168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260811 03:05:33.874241 5168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "8f1a0277ffdc46fa9d17fa20917c2ae3"
format_stamp: "Formatted at 2026-08-11 03:05:33 on dist-test-slave-8tdl"
server_key: "8a0f363d3ffbfe9d80fb8eee3f4a219a"
server_key_iv: "a6814b4b0cacdae7273c7b99414f46f3"
server_key_version: "encryptionkey@0"
I20260811 03:05:33.874460 5168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:33.902933 5168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:33.907711 5168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:33.907984 5168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:33.908957 5168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:33.910404 5168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:33.910481 5168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:33.910568 5168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:33.910604 5168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:33.948643 5168 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:35033
I20260811 03:05:33.948695 5315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:35033 every 8 connection(s)
I20260811 03:05:33.950027 5168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:05:33.951872 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 5168
I20260811 03:05:33.952102 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:05:33.952989 28084 external_mini_cluster.cc:1428] Setting key a0251c1715d1d4b7aad1a4c415600bb0
Aug 11 03:05:33 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417533, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:05:33.978780 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:33.953110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:33435 (local address 127.27.109.62:34745)
0811 03:05:33.959157 (+ 6047us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:33.959166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:33.959709 (+ 543us) server_negotiation.cc:407] Connection header received
0811 03:05:33.961726 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:33.961732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:33.961832 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:33.961958 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:33.963687 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:33.964443 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:33.966169 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:33.966588 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:33.966818 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:33.971842 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:33.971848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:33.971851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:33.971890 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:33.974729 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:33.975833 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:33.975838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:33.975840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:33.975973 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:33.976524 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:33.976528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:33.976530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:33.976807 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:33.976922 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:33.978134 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0811 03:05:33.978389 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5887,"thread_start_us":5761,"threads_started":1}
I20260811 03:05:33.980366 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:33.953743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:33435)
0811 03:05:33.959542 (+ 5799us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:33.959588 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:33.961431 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:33.962220 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:33.962228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:33.962897 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:33.963509 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:33.963527 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:33.964622 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:33.964626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:33.966015 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:33.966033 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:33.966868 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:33.966877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:33.967162 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:33.967948 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:33.967984 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:33.971526 (+ 3542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:33.974916 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:33.974928 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:33.974950 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:33.975663 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:33.976100 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:33.976108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:33.976114 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:33.976388 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:33.976943 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:33.976953 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:33.977520 (+ 567us) client_negotiation.cc:769] Sending connection context
0811 03:05:33.977856 (+ 336us) client_negotiation.cc:240] Negotiation successful
0811 03:05:33.978562 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5537,"thread_start_us":5290,"threads_started":1}
I20260811 03:05:33.982177 5316 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34745
I20260811 03:05:33.982569 5316 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:33.983556 5316 heartbeater.cc:511] Master 127.27.109.62:34745 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260811 03:05:33.986218 5098 ts_manager.cc:194] Registered new tserver with Master: 8f1a0277ffdc46fa9d17fa20917c2ae3 (127.27.109.1:35033)
I20260811 03:05:33.988225 5098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:33435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:05:34.026638 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34745
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:34.393314 5323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:34.393654 5323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:34.393764 5323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:34.402206 5323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:34.402345 5323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:34.402464 5323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:34.412864 5323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:34.414299 5323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:34.415700 5323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:34.429466 5329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:34.429435 5330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:34.429582 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:34.430770 5323 server_base.cc:1034] running on GCE node
I20260811 03:05:34.431432 5323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:34.432417 5323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:34.433686 5323 hybrid_clock.cc:648] HybridClock initialized: now 1786417534433638 us; error 47 us; skew 500 ppm
I20260811 03:05:34.434044 5323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:34 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417534, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:34.437548 5323 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:05:34.439318 5323 webserver.cc:460] Webserver started at http://127.27.109.2:45265/ using document root <none> and password file <none>
I20260811 03:05:34.440070 5323 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:34.440204 5323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:34.440543 5323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:34.443732 5323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "085080dbf2de45ff8522158b54f2aec4"
format_stamp: "Formatted at 2026-08-11 03:05:34 on dist-test-slave-8tdl"
server_key: "e3b191f2c0f02d4dd16a4722d81df64e"
server_key_iv: "2b1887518334e6a4fdd93f86a7c4430e"
server_key_version: "encryptionkey@0"
I20260811 03:05:34.444582 5323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "085080dbf2de45ff8522158b54f2aec4"
format_stamp: "Formatted at 2026-08-11 03:05:34 on dist-test-slave-8tdl"
server_key: "e3b191f2c0f02d4dd16a4722d81df64e"
server_key_iv: "2b1887518334e6a4fdd93f86a7c4430e"
server_key_version: "encryptionkey@0"
I20260811 03:05:34.450166 5323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:05:34.454073 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:34.455629 5323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:34.455830 5323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "085080dbf2de45ff8522158b54f2aec4"
format_stamp: "Formatted at 2026-08-11 03:05:34 on dist-test-slave-8tdl"
server_key: "e3b191f2c0f02d4dd16a4722d81df64e"
server_key_iv: "2b1887518334e6a4fdd93f86a7c4430e"
server_key_version: "encryptionkey@0"
I20260811 03:05:34.456068 5323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:34.474857 5323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:34.479797 5323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:34.480121 5323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:34.481189 5323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:34.482729 5323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:34.482841 5323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:34.483007 5323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:34.483084 5323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:34.518081 5323 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:35395
I20260811 03:05:34.518129 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:35395 every 8 connection(s)
I20260811 03:05:34.519532 5323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:34.528441 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 5323
I20260811 03:05:34.528661 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:05:34.529145 28084 external_mini_cluster.cc:1428] Setting key c99bbbd8eada0767fb406d08f237dc64
Aug 11 03:05:34 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417534, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:05:34.542848 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:34.522518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:57905 (local address 127.27.109.62:34745)
0811 03:05:34.523023 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:34.523030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:34.524111 (+ 1081us) server_negotiation.cc:407] Connection header received
0811 03:05:34.525994 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:34.525999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:34.526071 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:34.526172 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:34.527904 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:34.529010 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:34.531138 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:34.531703 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:34.531843 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:34.536630 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:34.536635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:34.536639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:34.536674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:34.539573 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:34.540356 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:34.540369 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:34.540372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:34.540499 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:34.540969 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:34.540973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:34.540975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:34.541263 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:34.541426 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:34.542175 (+ 749us) server_negotiation.cc:299] Negotiation successful
0811 03:05:34.542414 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
I20260811 03:05:34.544019 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:34.523166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:57905)
0811 03:05:34.523929 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:34.523974 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:34.525716 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:34.526431 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:34.526440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:34.527093 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:34.527695 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:34.527714 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:34.529220 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:34.529226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:34.530942 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:34.530955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:34.531985 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:34.532000 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:34.532370 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:34.533162 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:34.533205 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:34.536352 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:34.539740 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:34.539750 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:34.539776 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:34.540217 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:34.540619 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:34.540624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:34.540628 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:34.540830 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:34.541408 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:34.541417 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:34.541785 (+ 368us) client_negotiation.cc:769] Sending connection context
0811 03:05:34.542069 (+ 284us) client_negotiation.cc:240] Negotiation successful
0811 03:05:34.542666 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":304,"threads_started":1}
I20260811 03:05:34.545706 5464 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34745
I20260811 03:05:34.546085 5464 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:34.547091 5464 heartbeater.cc:511] Master 127.27.109.62:34745 requested a full tablet report, sending...
I20260811 03:05:34.549105 5098 ts_manager.cc:194] Registered new tserver with Master: 085080dbf2de45ff8522158b54f2aec4 (127.27.109.2:35395)
I20260811 03:05:34.550486 5098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:57905
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:05:34.602064 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34745
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:34.963367 5470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:34.963729 5470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:34.963897 5470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:34.971736 5470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:34.971908 5470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:34.972095 5470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:34.982616 5470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:34.984102 5470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:34.985605 5470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 03:05:34.992003 5316 heartbeater.cc:503] Master 127.27.109.62:34745 was elected leader, sending a full tablet report...
W20260811 03:05:35.000586 5475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:35.000607 5478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:35.000778 5476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:35.001760 5470 server_base.cc:1034] running on GCE node
I20260811 03:05:35.002521 5470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:35.003695 5470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:35.004999 5470 hybrid_clock.cc:648] HybridClock initialized: now 1786417535004915 us; error 79 us; skew 500 ppm
I20260811 03:05:35.005368 5470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:35 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417535, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:35.009375 5470 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:05:35.011358 5470 webserver.cc:460] Webserver started at http://127.27.109.3:40841/ using document root <none> and password file <none>
I20260811 03:05:35.012200 5470 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:35.012356 5470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:35.012734 5470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:35.016160 5470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "a840068ca36742fb930404a5d4287751"
format_stamp: "Formatted at 2026-08-11 03:05:35 on dist-test-slave-8tdl"
server_key: "7a18c915aa3ffa4798d2e4c247676156"
server_key_iv: "fd8e6216ea122b6ea9573cc8e14346b5"
server_key_version: "encryptionkey@0"
I20260811 03:05:35.017091 5470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "a840068ca36742fb930404a5d4287751"
format_stamp: "Formatted at 2026-08-11 03:05:35 on dist-test-slave-8tdl"
server_key: "7a18c915aa3ffa4798d2e4c247676156"
server_key_iv: "fd8e6216ea122b6ea9573cc8e14346b5"
server_key_version: "encryptionkey@0"
I20260811 03:05:35.023578 5470 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260811 03:05:35.027715 5487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:35.029341 5470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:05:35.029583 5470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "a840068ca36742fb930404a5d4287751"
format_stamp: "Formatted at 2026-08-11 03:05:35 on dist-test-slave-8tdl"
server_key: "7a18c915aa3ffa4798d2e4c247676156"
server_key_iv: "fd8e6216ea122b6ea9573cc8e14346b5"
server_key_version: "encryptionkey@0"
I20260811 03:05:35.029815 5470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:35.048436 5470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:35.053284 5470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:35.053617 5470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:35.054664 5470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:35.056195 5470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:35.056308 5470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:35.056494 5470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:35.056571 5470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:35.093961 5470 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:45353
I20260811 03:05:35.094017 5610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:45353 every 8 connection(s)
I20260811 03:05:35.095712 5470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:35.102862 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 5470
I20260811 03:05:35.103088 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:05:35.103571 28084 external_mini_cluster.cc:1428] Setting key 5032e33f8015d06db2f8cee86d4d4b7c
Aug 11 03:05:35 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417535, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:05:35.118402 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:35.098774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:54777 (local address 127.27.109.62:34745)
0811 03:05:35.099396 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:35.099400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:35.100147 (+ 747us) server_negotiation.cc:407] Connection header received
0811 03:05:35.102034 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:35.102040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:35.102133 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:35.102283 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:35.104074 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.105162 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.107158 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.107582 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.107721 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:35.111871 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:35.111877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:35.111880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:35.111919 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:35.114997 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:35.115846 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:35.115851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:35.115853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:35.115988 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:35.116416 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:35.116419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:35.116421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:35.116689 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:35.116808 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:35.117790 (+ 982us) server_negotiation.cc:299] Negotiation successful
0811 03:05:35.118026 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":389,"threads_started":1}
I20260811 03:05:35.119825 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:35.099226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:54777)
0811 03:05:35.099971 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:35.100022 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:35.101748 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:35.102521 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:35.102530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:35.103247 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:35.103889 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:35.103909 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.105324 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.105328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:35.107014 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:35.107027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.107830 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.107837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:35.108089 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:35.108808 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:35.108836 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:35.111592 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:35.115170 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:35.115198 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:35.115214 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:35.115728 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:35.116092 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:05:35.116096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:05:35.116098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:05:35.116296 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:05:35.116810 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:35.116816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:35.117282 (+ 466us) client_negotiation.cc:769] Sending connection context
0811 03:05:35.117649 (+ 367us) client_negotiation.cc:240] Negotiation successful
0811 03:05:35.118259 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":292,"threads_started":1}
I20260811 03:05:35.121426 5611 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34745
I20260811 03:05:35.121774 5611 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:35.122633 5611 heartbeater.cc:511] Master 127.27.109.62:34745 requested a full tablet report, sending...
I20260811 03:05:35.124343 5098 ts_manager.cc:194] Registered new tserver with Master: a840068ca36742fb930404a5d4287751 (127.27.109.3:45353)
I20260811 03:05:35.125324 5098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:54777
I20260811 03:05:35.132817 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
Entry for principal oryx/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab.
WARNING: no policy specified for oryx/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
Entry for principal oryx/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab.
WARNING: no policy specified for oryx/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
Entry for principal oryx/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab.
I20260811 03:05:35.325417 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5168
Aug 11 03:05:35 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:05:35.344069 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:35.326883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.27.109.1:35033)
0811 03:05:35.327543 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:35.327551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:35.327579 (+ 28us) server_negotiation.cc:407] Connection header received
0811 03:05:35.327654 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:35.327660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:35.327883 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:35.328056 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:35.329867 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.331303 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.332730 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.333575 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.333692 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:35.338352 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:35.338366 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:35.338382 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:35.338422 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:35.341335 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:35.342043 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:35.342051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:35.342053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:35.342195 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:35.342545 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:35.342548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:35.342550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:35.343039 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:35.343291 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:35.343556 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 03:05:35.343767 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":314,"threads_started":1}
I20260811 03:05:35.345450 5311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:35.432343 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 5168
I20260811 03:05:35.453765 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5323
Aug 11 03:05:35 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:05:35.471218 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:35.454767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.27.109.2:35395)
0811 03:05:35.455307 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:35.455315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:35.455339 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:05:35.455405 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:35.455412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:35.455678 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:35.455816 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:35.457123 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.458147 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.459529 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.460099 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.460232 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:35.464650 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:35.464663 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:35.464679 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:35.464716 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:35.468053 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:35.468767 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:35.468774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:35.468777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:35.468909 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:35.469359 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:35.469363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:35.469365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:35.469736 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:35.470084 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:35.470396 (+ 312us) server_negotiation.cc:299] Negotiation successful
0811 03:05:35.470749 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":275,"threads_started":1}
I20260811 03:05:35.472671 5457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:35.558992 5464 heartbeater.cc:503] Master 127.27.109.62:34745 was elected leader, sending a full tablet report...
I20260811 03:05:35.560210 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 5323
W20260811 03:05:35.580776 5082 connection.cc:535] server connection from 127.27.109.2:57905 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 03:05:35.580935 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5470
Aug 11 03:05:35 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:05:35.599503 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:35.582044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.27.109.3:45353)
0811 03:05:35.582650 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:35.582658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:35.582753 (+ 95us) server_negotiation.cc:407] Connection header received
0811 03:05:35.582822 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:35.582828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:35.583082 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:35.583218 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:35.584595 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.585629 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.587049 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:35.587669 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:35.587789 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:35.592921 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:35.592939 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:35.592955 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:35.593004 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:35.596085 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:35.596837 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:35.596844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:35.596846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:35.596989 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:35.597408 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:35.597412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:35.597414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:35.598012 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:35.598288 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:35.598736 (+ 448us) server_negotiation.cc:299] Negotiation successful
0811 03:05:35.599039 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":229,"threads_started":1}
I20260811 03:05:35.600967 5606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:35.692785 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 5470
I20260811 03:05:35.713387 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5062
I20260811 03:05:35.714880 5131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:35.828264 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 5062
I20260811 03:05:35.854322 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34745
--webserver_interface=127.27.109.62
--webserver_port=34769
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:05:36.223176 5633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:36.223484 5633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:36.223577 5633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:36.231453 5633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:36.231594 5633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:36.231644 5633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:36.231709 5633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:36.231756 5633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:36.242164 5633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=34769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:36.243597 5633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:36.245020 5633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:36.257725 5639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:36.257754 5638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:36.257969 5641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:36.258841 5633 server_base.cc:1034] running on GCE node
I20260811 03:05:36.259690 5633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:36.261220 5633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:36.262503 5633 hybrid_clock.cc:648] HybridClock initialized: now 1786417536262451 us; error 43 us; skew 500 ppm
I20260811 03:05:36.262961 5633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:36 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:36.266407 5633 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:05:36.268213 5633 webserver.cc:460] Webserver started at http://127.27.109.62:34769/ using document root <none> and password file <none>
I20260811 03:05:36.269402 5633 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:36.269543 5633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:36.276384 5633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 03:05:36.279516 5649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:36.281000 5633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:36.281210 5633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "223d965189844e348507d547bcb57775"
format_stamp: "Formatted at 2026-08-11 03:05:33 on dist-test-slave-8tdl"
server_key: "51683eca1d72aa8d15eb62d3f928a9da"
server_key_iv: "c25f731ec67214132493af370c0e1564"
server_key_version: "encryptionkey@0"
I20260811 03:05:36.282297 5633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:36.294909 5633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:36.299726 5633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:36.300024 5633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:36.320659 5633 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:34745
I20260811 03:05:36.320683 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:34745 every 8 connection(s)
I20260811 03:05:36.322377 5633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:36.325084 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 5633
I20260811 03:05:36.325956 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:35033
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:34745
--webserver_port=43389
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.1
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:05:36.329945 5706 sys_catalog.cc:263] Verifying existing consensus state
I20260811 03:05:36.333542 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Bootstrap starting.
I20260811 03:05:36.353994 5706 log.cc:824] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:36.363868 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 03:05:36.364462 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Bootstrap complete.
I20260811 03:05:36.373977 5706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } }
I20260811 03:05:36.375005 5706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223d965189844e348507d547bcb57775, State: Initialized, Role: FOLLOWER
I20260811 03:05:36.375756 5706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } }
I20260811 03:05:36.376049 5706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:36.376196 5706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:36.376405 5706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 1 FOLLOWER]: Advancing to term 2
I20260811 03:05:36.380548 5706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } }
I20260811 03:05:36.381088 5706 leader_election.cc:302] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 223d965189844e348507d547bcb57775; no voters:
I20260811 03:05:36.381776 5706 leader_election.cc:288] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 03:05:36.381923 5711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 03:05:36.383214 5711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [term 2 LEADER]: Becoming Leader. State: Replica: 223d965189844e348507d547bcb57775, State: Running, Role: LEADER
I20260811 03:05:36.383867 5711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } }
I20260811 03:05:36.384212 5706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:36.387156 5713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 223d965189844e348507d547bcb57775. Latest consensus state: current_term: 2 leader_uuid: "223d965189844e348507d547bcb57775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } } }
I20260811 03:05:36.387787 5713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:36.387127 5712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "223d965189844e348507d547bcb57775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223d965189844e348507d547bcb57775" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34745 } } }
I20260811 03:05:36.388495 5712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775 [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:36.399503 5718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:36.406515 5718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:36.408177 5718 catalog_manager.cc:1259] Loaded cluster ID: a6c6a6882b6549e7b878697d54ddc466
I20260811 03:05:36.408419 5718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:36.412925 5718 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:36.414649 5718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 223d965189844e348507d547bcb57775: Loaded TSK: 0
I20260811 03:05:36.416376 5718 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 03:05:36.700984 5709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:36.701406 5709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:36.701575 5709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:36.709529 5709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:36.709710 5709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:36.709895 5709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:05:36.720278 5709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.1.keytab
--principal=oryx/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:35033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=43389
--tserver_master_addrs=127.27.109.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:36.721849 5709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:36.723305 5709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:36.738832 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:36.739037 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:36.739658 5737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:36.740988 5709 server_base.cc:1034] running on GCE node
I20260811 03:05:36.741825 5709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:36.743233 5709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:36.744516 5709 hybrid_clock.cc:648] HybridClock initialized: now 1786417536744427 us; error 62 us; skew 500 ppm
I20260811 03:05:36.744900 5709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:36 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417536, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:36.749392 5709 init.cc:377] Logged in from keytab as oryx/127.27.109.1@KRBTEST.COM (short username oryx)
I20260811 03:05:36.751852 5709 webserver.cc:460] Webserver started at http://127.27.109.1:43389/ using document root <none> and password file <none>
I20260811 03:05:36.752837 5709 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:36.753003 5709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:36.760679 5709 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:05:36.764284 5745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:36.766157 5709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:05:36.766355 5709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "8f1a0277ffdc46fa9d17fa20917c2ae3"
format_stamp: "Formatted at 2026-08-11 03:05:33 on dist-test-slave-8tdl"
server_key: "8a0f363d3ffbfe9d80fb8eee3f4a219a"
server_key_iv: "a6814b4b0cacdae7273c7b99414f46f3"
server_key_version: "encryptionkey@0"
I20260811 03:05:36.767809 5709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:36.801270 5709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:36.806573 5709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:36.806999 5709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:36.808145 5709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:36.809707 5709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:36.809808 5709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:36.809906 5709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:36.809970 5709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:36.848449 5709 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:35033
I20260811 03:05:36.848554 5871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:35033 every 8 connection(s)
I20260811 03:05:36.850157 5709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:05:36.858435 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 5709
I20260811 03:05:36.859200 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:35395
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:34745
--webserver_port=45265
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.2
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:05:36 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:36 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 03:05:37.227178 5876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:37.227478 5876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:37.227625 5876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:37.235373 5876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:37.235504 5876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:37.235684 5876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:37.246006 5876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.2.keytab
--principal=oryx/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:35395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=45265
--tserver_master_addrs=127.27.109.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:37.247581 5876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:37.249143 5876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:37.268882 5885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:37.269457 5888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:37.268841 5884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:37.269796 5876 server_base.cc:1034] running on GCE node
I20260811 03:05:37.270552 5876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:37.271685 5876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:37.273267 5876 hybrid_clock.cc:648] HybridClock initialized: now 1786417537273162 us; error 89 us; skew 500 ppm
I20260811 03:05:37.273617 5876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:37 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417537, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:37.277751 5876 init.cc:377] Logged in from keytab as oryx/127.27.109.2@KRBTEST.COM (short username oryx)
I20260811 03:05:37.280085 5876 webserver.cc:460] Webserver started at http://127.27.109.2:45265/ using document root <none> and password file <none>
I20260811 03:05:37.280864 5876 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:37.281036 5876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:37.287043 5876 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:05:37.290560 5898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:37.292415 5876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:05:37.292636 5876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "085080dbf2de45ff8522158b54f2aec4"
format_stamp: "Formatted at 2026-08-11 03:05:34 on dist-test-slave-8tdl"
server_key: "e3b191f2c0f02d4dd16a4722d81df64e"
server_key_iv: "2b1887518334e6a4fdd93f86a7c4430e"
server_key_version: "encryptionkey@0"
I20260811 03:05:37.293778 5876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:37.317755 5876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:37.322999 5876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:37.323352 5876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:37.324667 5876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:37.326455 5876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:37.326592 5876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:37.326789 5876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:37.326896 5876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:37.375871 5876 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:35395
I20260811 03:05:37.375950 6028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:35395 every 8 connection(s)
I20260811 03:05:37.377650 5876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:37.382108 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 5876
I20260811 03:05:37.382879 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:45353
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:34745
--webserver_port=40841
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.109.3
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:05:37 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:37 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 03:05:38.106492 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:38.106940 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:38.107153 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:38.118096 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:38.118259 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:38.118520 6033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:38.133944 6033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/krb5kdc/oryx_127.27.109.3.keytab
--principal=oryx/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:45353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=40841
--tserver_master_addrs=127.27.109.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:38.135794 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:38.137537 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:38.149299 5875 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.284s user 0.006s sys 0.009s
W20260811 03:05:38.151901 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:36.853656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:39275 (local address 127.27.109.62:34745)
0811 03:05:36.854597 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:36.854614 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:36.854857 (+ 243us) server_negotiation.cc:407] Connection header received
0811 03:05:36.857462 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:36.857482 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:36.857880 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:36.858268 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:36.860071 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:36.861360 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:36.863167 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:36.863909 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:36.864181 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.150970 (+1286789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:39275: BlockingRecv error: recv got EOF from 127.27.109.1:39275 (error 108)
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":355,"threads_started":1}
I20260811 03:05:38.151925 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:36.853711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:39275)
0811 03:05:36.854615 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:36.854667 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:36.856825 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:36.858477 (+ 1652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:36.858490 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:36.859191 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:36.859830 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:36.859848 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:36.861569 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:36.861573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:36.862999 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:36.863014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:36.864228 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:36.864237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:36.864606 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:36.865395 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:36.865426 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:38.150484 (+1285058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":365,"threads_started":1}
W20260811 03:05:38.152856 5872 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:34745 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.62:34745: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
W20260811 03:05:38.158371 6039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:38.159848 6040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 03:05:38.167544 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:38.167953 6033 server_base.cc:1034] running on GCE node
I20260811 03:05:38.169060 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:38.153590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:47241)
0811 03:05:38.153926 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:38.153958 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:38.154178 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:38.154903 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:38.154911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:38.155540 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:38.155833 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.155849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.157751 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.157759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.159421 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.159441 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.160902 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.160909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.161188 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.162092 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:38.162127 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:38.168692 (+ 6565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":54}
W20260811 03:05:38.169266 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:38.153804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:47241 (local address 127.27.109.62:34745)
0811 03:05:38.153961 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:38.153968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:38.154039 (+ 71us) server_negotiation.cc:407] Connection header received
0811 03:05:38.154357 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:38.154366 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:38.154467 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:38.154561 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:38.156079 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.157554 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.159621 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.160302 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.160465 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.168972 (+ 8507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:47241: BlockingRecv error: recv got EOF from 127.27.109.1:47241 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 03:05:38.169479 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:38.171363 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:38.172834 6033 hybrid_clock.cc:648] HybridClock initialized: now 1786417538172770 us; error 85 us; skew 500 ppm
I20260811 03:05:38.173233 6033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417538, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:05:38.177160 6033 init.cc:377] Logged in from keytab as oryx/127.27.109.3@KRBTEST.COM (short username oryx)
I20260811 03:05:38.182148 6033 webserver.cc:460] Webserver started at http://127.27.109.3:40841/ using document root <none> and password file <none>
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:38.183223 6033 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:38.183411 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:38.186216 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:38.170110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:46775)
0811 03:05:38.170379 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:38.170400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:38.170549 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:38.170985 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:38.170993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:38.171461 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:38.171719 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.171735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.173996 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.174006 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.176632 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.176660 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.178745 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.178756 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.179111 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.180097 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:38.180134 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:38.185859 (+ 5725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153}
W20260811 03:05:38.186749 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:38.170197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:46775 (local address 127.27.109.62:34745)
0811 03:05:38.170382 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:38.170389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:38.170457 (+ 68us) server_negotiation.cc:407] Connection header received
0811 03:05:38.170721 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:38.170726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:38.170823 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:38.170942 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:38.171941 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.173704 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.177066 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.178095 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.178266 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.186326 (+ 8060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:46775: BlockingRecv error: recv got EOF from 127.27.109.1:46775 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260811 03:05:38.186828 5872 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 03:05:38.191928 6033 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260811 03:05:38.195807 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:38.198064 6033 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260811 03:05:38.198304 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "a840068ca36742fb930404a5d4287751"
format_stamp: "Formatted at 2026-08-11 03:05:35 on dist-test-slave-8tdl"
server_key: "7a18c915aa3ffa4798d2e4c247676156"
server_key_iv: "fd8e6216ea122b6ea9573cc8e14346b5"
server_key_version: "encryptionkey@0"
I20260811 03:05:38.199663 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:38.215560 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:38.220861 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:38.221208 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:38.222573 6033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:38.224417 6033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:38.224570 6033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:38.224727 6033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:38.224826 6033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:38.271982 6033 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:45353
I20260811 03:05:38.272043 6177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:45353 every 8 connection(s)
I20260811 03:05:38.273797 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:38.282619 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 6033
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:38.320817 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:38.283989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.27.109.62:34745)
0811 03:05:38.296296 (+ 12307us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:38.296305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:38.296440 (+ 135us) server_negotiation.cc:407] Connection header received
0811 03:05:38.296705 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:38.296713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:38.296813 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:38.296958 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:38.300468 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.302127 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.305299 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.306575 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.306772 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.312185 (+ 5413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:38.312203 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:38.312234 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:38.312307 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:38.316698 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:38.317635 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:38.317644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:38.317653 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:38.317866 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:05:38.318341 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:05:38.318346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:05:38.318348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:05:38.319006 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:38.319411 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:38.319921 (+ 510us) server_negotiation.cc:299] Negotiation successful
0811 03:05:38.320375 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12140,"thread_start_us":12306,"threads_started":1}
W20260811 03:05:38.731611 6032 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.317s user 0.005s sys 0.016s
I20260811 03:05:38.733598 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:37.382487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:42799)
0811 03:05:37.397984 (+ 15497us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:37.398081 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:37.402341 (+ 4260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:37.403722 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:37.403743 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:37.404968 (+ 1225us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:37.406260 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:37.406303 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:37.408254 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:37.408264 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:37.411531 (+ 3267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:37.411559 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:37.413077 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:37.413088 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:37.413568 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:37.414659 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:37.414743 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:38.732655 (+1317912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14921,"thread_start_us":14593,"threads_started":1}
W20260811 03:05:38.733748 6031 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:37.381917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:42799 (local address 127.27.109.62:34745)
0811 03:05:37.398513 (+ 16596us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:37.398519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:37.398542 (+ 23us) server_negotiation.cc:407] Connection header received
0811 03:05:37.402937 (+ 4395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:37.402946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:37.403078 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:37.403219 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:37.406635 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:37.408007 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:37.411847 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:37.412674 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:37.412868 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.733184 (+1320316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:42799: BlockingRecv error: recv got EOF from 127.27.109.2:42799 (error 108)
Metrics: {"server-negotiator.queue_time_us":16452,"thread_start_us":16599,"threads_started":1}
W20260811 03:05:38.735041 6029 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:34745 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.62:34745: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:38.750658 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:38.736100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:33243)
0811 03:05:38.736435 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:38.736469 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:38.736690 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:38.737384 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:38.737392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:38.738249 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:38.738650 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.738671 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.740177 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.740187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.742980 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.743016 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.744338 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.744351 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.744703 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.745738 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:38.745765 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:38.750335 (+ 4570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260811 03:05:38.750939 6031 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:38.736247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:33243 (local address 127.27.109.62:34745)
0811 03:05:38.736463 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:38.736468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:38.736574 (+ 106us) server_negotiation.cc:407] Connection header received
0811 03:05:38.736953 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:38.736961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:38.737142 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:38.737290 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:38.739032 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.739968 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.743265 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.743984 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.744166 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.750637 (+ 6471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:33243: BlockingRecv error: recv got EOF from 127.27.109.2:33243 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:38 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:38.762076 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:38.751617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:57585)
0811 03:05:38.751790 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:38.751811 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:38.751926 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:38.752715 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:38.752720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:38.753144 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:38.753321 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.753331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.754515 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.754519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.755590 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.755602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.756591 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.756598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.756795 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.757634 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:38.757653 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:38.761749 (+ 4096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 03:05:38.762567 6029 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 03:05:38.762821 6031 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:38.752174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:57585 (local address 127.27.109.62:34745)
0811 03:05:38.752409 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:38.752413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:38.752430 (+ 17us) server_negotiation.cc:407] Connection header received
0811 03:05:38.752485 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:38.752488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:38.752547 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:38.752666 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:38.753480 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.754338 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.755750 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.756422 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.756567 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.761990 (+ 5423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:57585: BlockingRecv error: recv got EOF from 127.27.109.2:57585 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 11 03:05:39 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:39 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:39.208295 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:39.187559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:52923)
0811 03:05:39.188331 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:39.188348 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:39.188477 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:39.189088 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:39.189091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:39.189560 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:39.189738 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:39.189748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.192145 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.192150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:39.193259 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:39.193275 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.195428 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.195438 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:39.195651 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.201348 (+ 5697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:39.201471 (+ 123us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:39.207854 (+ 6383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":448,"threads_started":1}
W20260811 03:05:39.208841 6031 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:39.187642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:52923 (local address 127.27.109.62:34745)
0811 03:05:39.187976 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:39.187988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:39.188459 (+ 471us) server_negotiation.cc:407] Connection header received
0811 03:05:39.188648 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:39.188658 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:39.188849 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:39.189046 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:39.189945 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.191870 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.193472 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.194660 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.195177 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.208415 (+ 13238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:52923: BlockingRecv error: recv got EOF from 127.27.109.1:52923 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260811 03:05:39.565160 6180 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.272s user 0.004s sys 0.015s
W20260811 03:05:39.566458 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:38.277878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:35217 (local address 127.27.109.62:34745)
0811 03:05:38.278216 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:38.278228 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:38.279461 (+ 1233us) server_negotiation.cc:407] Connection header received
0811 03:05:38.281608 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:38.281618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:38.281752 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:38.281979 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:38.284628 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.286238 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.290055 (+ 3817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.291208 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.291356 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.566234 (+1274878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:35217: BlockingRecv error: recv got EOF from 127.27.109.3:35217 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260811 03:05:39.567003 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:38.278348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:35217)
0811 03:05:38.279278 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:38.279327 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:38.281292 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:38.282381 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:38.282437 (+ 56us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:38.283364 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:38.284284 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.284325 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.286614 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.286624 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.289785 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:38.289810 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:38.291892 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:38.291904 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:38.292396 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:38.293536 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:38.293592 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:39.565970 (+1272378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":426,"threads_started":1}
W20260811 03:05:39.568066 6178 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:34745 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.109.62:34745: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Aug 11 03:05:39 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:39 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:39.579061 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:39.568847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:49845)
0811 03:05:39.569051 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:39.569070 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:39.569219 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:39.569524 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:39.569527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:39.569934 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:39.570179 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:39.570190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.571390 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.571395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:39.572481 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:39.572496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.573369 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.573375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:39.573511 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.574306 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:39.574326 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:39.578806 (+ 4480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260811 03:05:39.579355 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:39.568979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:49845 (local address 127.27.109.62:34745)
0811 03:05:39.569210 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:39.569214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:39.569233 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:05:39.569325 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:39.569328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:39.569389 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:39.569478 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:39.570339 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.571242 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.572629 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.573195 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.573296 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.579097 (+ 5801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:49845: BlockingRecv error: recv got EOF from 127.27.109.3:49845 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Aug 11 03:05:39 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:39 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:39.589723 6180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:39.579919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:48305)
0811 03:05:39.580130 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:39.580149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:39.580243 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:39.580600 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:39.580605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:39.580930 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:39.581150 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:39.581160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.582244 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.582247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:39.583384 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:39.583397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.584138 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.584175 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:39.584397 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.585027 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:39.585047 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:39.589507 (+ 4460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260811 03:05:39.590039 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:39.580155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:48305 (local address 127.27.109.62:34745)
0811 03:05:39.580314 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:39.580318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:39.580334 (+ 16us) server_negotiation.cc:407] Connection header received
0811 03:05:39.580399 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:39.580402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:39.580458 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:39.580543 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:39.581290 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.582115 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.583522 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.583939 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.584040 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.589766 (+ 5726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:48305: BlockingRecv error: recv got EOF from 127.27.109.3:48305 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260811 03:05:39.590181 6178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 03:05:39 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:39 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:39.774222 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:39.763440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:45619)
0811 03:05:39.764122 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:39.764145 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:39.764289 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:39.764711 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:39.764716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:39.765215 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:39.765377 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:39.765386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.766486 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.766491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:39.767722 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:39.767737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.768652 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.768659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:39.768859 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.769554 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:39.769574 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:39.773785 (+ 4211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":389,"threads_started":1}
W20260811 03:05:39.774334 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:39.763563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:45619 (local address 127.27.109.62:34745)
0811 03:05:39.763786 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:39.763791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:39.764251 (+ 460us) server_negotiation.cc:407] Connection header received
0811 03:05:39.764479 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:39.764484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:39.764555 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:39.764650 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:39.765539 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.766306 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.767860 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:39.768448 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:39.768609 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:39.774068 (+ 5459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:45619: BlockingRecv error: recv got EOF from 127.27.109.2:45619 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 11 03:05:40 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:40 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:40.220432 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:40.209686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:44281)
0811 03:05:40.210273 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:40.210292 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:40.210423 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:40.210921 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:40.210926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:40.211406 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:40.211610 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:40.211620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.213150 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.213157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:40.214348 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:40.214362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.215157 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.215162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:40.215290 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:40.216054 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:40.216073 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:40.220163 (+ 4090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":333,"threads_started":1}
W20260811 03:05:40.220715 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:40.209786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44281 (local address 127.27.109.62:34745)
0811 03:05:40.210026 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:40.210031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:40.210394 (+ 363us) server_negotiation.cc:407] Connection header received
0811 03:05:40.210614 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:40.210619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:40.210725 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:40.210833 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:40.211838 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.212958 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.214488 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.214991 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.215101 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:40.220408 (+ 5307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:44281: BlockingRecv error: recv got EOF from 127.27.109.1:44281 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 11 03:05:40 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:40 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:40.601565 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:40.591004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:56325)
0811 03:05:40.591699 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:40.591716 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:40.591841 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:40.592264 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:40.592268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:40.592775 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:40.592961 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:40.592972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.594123 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.594129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:40.595199 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:40.595213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.596009 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.596017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:40.596187 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:40.596958 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:40.596980 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:40.601297 (+ 4317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":440,"threads_started":1}
W20260811 03:05:40.601946 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:40.591069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:56325 (local address 127.27.109.62:34745)
0811 03:05:40.591273 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:40.591278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:40.591801 (+ 523us) server_negotiation.cc:407] Connection header received
0811 03:05:40.591956 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:40.591960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:40.592103 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:40.592198 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:40.593150 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.593964 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.595352 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.595772 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.595942 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:40.601588 (+ 5646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:56325: BlockingRecv error: recv got EOF from 127.27.109.3:56325 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 11 03:05:40 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:40 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:40.786146 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:40.775337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:52451)
0811 03:05:40.775928 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:40.775949 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:40.776083 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:40.776496 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:40.776501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:40.776992 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:40.777238 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:40.777252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.778577 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.778581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:40.779631 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:40.779657 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.780387 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.780392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:40.780540 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:40.781272 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:40.781293 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:40.785838 (+ 4545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":338,"threads_started":1}
W20260811 03:05:40.786301 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:40.775473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:52451 (local address 127.27.109.62:34745)
0811 03:05:40.775659 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:40.775664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:40.776039 (+ 375us) server_negotiation.cc:407] Connection header received
0811 03:05:40.776198 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:40.776202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:40.776266 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:40.776356 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:40.777436 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.778406 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.779802 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:40.780224 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:40.780327 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:40.786068 (+ 5741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:52451: BlockingRecv error: recv got EOF from 127.27.109.2:52451 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 11 03:05:41 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:41 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:41.232398 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:41.221631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:34515)
0811 03:05:41.222308 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:41.222326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:41.222506 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:41.222935 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:41.222942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:41.223383 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:41.223592 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:41.223602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.224799 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.224806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:41.225863 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:41.225877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.226783 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.226790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:41.227022 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:41.227649 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:41.227669 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:41.232009 (+ 4340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":421,"threads_started":1}
W20260811 03:05:41.232704 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:41.221855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:34515 (local address 127.27.109.62:34745)
0811 03:05:41.222135 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:41.222144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:41.222426 (+ 282us) server_negotiation.cc:407] Connection header received
0811 03:05:41.222651 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:41.222657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:41.222767 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:41.222888 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:41.223776 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.224628 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.226066 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.226565 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.226748 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:41.232343 (+ 5595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:34515: BlockingRecv error: recv got EOF from 127.27.109.1:34515 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 11 03:05:41 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:41 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:41.613050 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:41.602574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:37333)
0811 03:05:41.603250 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:41.603267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:41.603388 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:41.603744 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:41.603749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:41.604389 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:41.604665 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:41.604680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.606169 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.606174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:41.607169 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:41.607181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.607961 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.607967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:41.608163 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:41.608817 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:41.608836 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:41.612662 (+ 3826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":455,"threads_started":1}
W20260811 03:05:41.613245 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:41.602909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:37333 (local address 127.27.109.62:34745)
0811 03:05:41.603122 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:41.603127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:41.603345 (+ 218us) server_negotiation.cc:407] Connection header received
0811 03:05:41.603505 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:41.603509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:41.603575 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:41.603703 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:41.604841 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.605841 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.607327 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.607783 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.607924 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:41.612884 (+ 4960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:37333: BlockingRecv error: recv got EOF from 127.27.109.3:37333 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 11 03:05:41 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:41 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:41.797807 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:41.787196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:47063)
0811 03:05:41.787804 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:41.787821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:41.788020 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:41.788389 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:41.788393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:41.788775 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:41.789013 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:41.789024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.790228 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.790232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:41.791196 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:41.791209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.792231 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.792236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:41.792371 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:41.793037 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:41.793057 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:41.797395 (+ 4338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":334,"threads_started":1}
W20260811 03:05:41.797957 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:41.787327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:47063 (local address 127.27.109.62:34745)
0811 03:05:41.787540 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:41.787548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:41.787997 (+ 449us) server_negotiation.cc:407] Connection header received
0811 03:05:41.788166 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:41.788170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:41.788243 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:41.788344 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:41.789181 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.790090 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.791348 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:41.792010 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:41.792125 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:41.797646 (+ 5521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:47063: BlockingRecv error: recv got EOF from 127.27.109.2:47063 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 11 03:05:42 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:42 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:42.244709 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:42.233771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:38225)
0811 03:05:42.234473 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:42.234491 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:42.234616 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:42.235244 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:42.235248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:42.235695 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:42.235880 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:42.235891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.237019 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.237024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:42.238338 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:42.238354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.239356 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.239361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:42.239554 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:42.240212 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:42.240231 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:42.244327 (+ 4096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":438,"threads_started":1}
W20260811 03:05:42.244910 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:42.233917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:38225 (local address 127.27.109.62:34745)
0811 03:05:42.234116 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:42.234121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:42.234622 (+ 501us) server_negotiation.cc:407] Connection header received
0811 03:05:42.234819 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:42.234824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:42.234915 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:42.235204 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:42.236100 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.236858 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.238510 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.239165 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.239281 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:42.244624 (+ 5343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:38225: BlockingRecv error: recv got EOF from 127.27.109.1:38225 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 11 03:05:42 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:42 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:42.624548 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:42.614380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:48353)
0811 03:05:42.615087 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:42.615115 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:42.615285 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:42.615680 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:42.615685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:42.616089 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:42.616263 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:42.616272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.617354 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.617360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:42.618287 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:42.618300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.619089 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.619095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:42.619294 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:42.619918 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:42.619936 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:42.624277 (+ 4341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":448,"threads_started":1}
W20260811 03:05:42.624809 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:42.614590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:48353 (local address 127.27.109.62:34745)
0811 03:05:42.614835 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:42.614917 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:42.615223 (+ 306us) server_negotiation.cc:407] Connection header received
0811 03:05:42.615444 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:42.615448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:42.615521 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:42.615635 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:42.616432 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.617204 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.618438 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.618933 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.619072 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:42.624533 (+ 5461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:48353: BlockingRecv error: recv got EOF from 127.27.109.3:48353 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 11 03:05:42 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:42 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:42.809737 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:42.799262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:56845)
0811 03:05:42.799926 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:42.799944 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:42.800069 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:42.800506 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:42.800514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:42.800921 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:42.801098 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:42.801107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.802311 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.802315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:42.803412 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:42.803426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.804315 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.804319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:42.804446 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:42.805203 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:42.805224 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:42.809470 (+ 4246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":394,"threads_started":1}
W20260811 03:05:42.809973 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:42.799331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:56845 (local address 127.27.109.62:34745)
0811 03:05:42.799575 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:42.799581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:42.800047 (+ 466us) server_negotiation.cc:407] Connection header received
0811 03:05:42.800232 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:42.800236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:42.800306 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:42.800412 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:42.801253 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.802163 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.803568 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:42.804152 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:42.804253 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:42.809664 (+ 5411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:56845: BlockingRecv error: recv got EOF from 127.27.109.2:56845 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 11 03:05:43 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:43 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:43.256029 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:43.245707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:47667)
0811 03:05:43.246321 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:43.246337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:43.246527 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:43.246916 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:43.246921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:43.247402 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:43.247575 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:43.247584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.248718 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.248722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:43.249693 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:43.249706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.250506 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.250512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:43.250752 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:43.251412 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:43.251430 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:43.255698 (+ 4268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":337,"threads_started":1}
W20260811 03:05:43.256233 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:43.245834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:47667 (local address 127.27.109.62:34745)
0811 03:05:43.246152 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:43.246160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:43.246429 (+ 269us) server_negotiation.cc:407] Connection header received
0811 03:05:43.246665 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:43.246702 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:43.246767 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:43.246878 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:43.247733 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.248523 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.249856 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.250327 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.250496 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:43.255923 (+ 5427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:47667: BlockingRecv error: recv got EOF from 127.27.109.1:47667 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 11 03:05:43 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:43 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:43.636422 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:43.625909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:46563)
0811 03:05:43.626521 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:43.626541 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:43.626663 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:43.627087 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:43.627090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:43.627540 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:43.627792 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:43.627811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.629295 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.629304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:43.630519 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:43.630533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.631315 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.631320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:43.631491 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:43.632127 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:43.632147 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:43.636125 (+ 3978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":333,"threads_started":1}
W20260811 03:05:43.636602 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:43.625961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:46563 (local address 127.27.109.62:34745)
0811 03:05:43.626190 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:43.626195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:43.626633 (+ 438us) server_negotiation.cc:407] Connection header received
0811 03:05:43.626818 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:43.626822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:43.626889 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:43.627031 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:43.627985 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.629094 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.630669 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.631165 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.631266 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:43.636370 (+ 5104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:46563: BlockingRecv error: recv got EOF from 127.27.109.3:46563 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 11 03:05:43 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:43 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:43.821517 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:43.810855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:38989)
0811 03:05:43.811599 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:43.811616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:43.811795 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:43.812190 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:43.812195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:43.812682 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:43.812839 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:43.812847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.813957 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.813973 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:43.815035 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:43.815049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.815887 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.815894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:43.816092 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:43.816764 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:43.816797 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:43.821145 (+ 4348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":435,"threads_started":1}
W20260811 03:05:43.821697 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:43.811101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:38989 (local address 127.27.109.62:34745)
0811 03:05:43.811395 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:43.811403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:43.811716 (+ 313us) server_negotiation.cc:407] Connection header received
0811 03:05:43.811941 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:43.811947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:43.812029 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:43.812140 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:43.813001 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.813795 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.815201 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:43.815717 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:43.815866 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:43.821388 (+ 5522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:38989: BlockingRecv error: recv got EOF from 127.27.109.2:38989 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 11 03:05:44 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:44 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:44.273901 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:44.257552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:42333)
0811 03:05:44.258197 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:44.258221 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:44.258441 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:44.258871 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:44.258877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:44.259445 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:44.259672 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:44.259682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.260702 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.260707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:44.261642 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:44.261654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.262449 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.262457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:44.262649 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:44.269093 (+ 6444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:44.269117 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:44.273659 (+ 4542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":382,"threads_started":1}
W20260811 03:05:44.274196 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:44.257660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:42333 (local address 127.27.109.62:34745)
0811 03:05:44.257876 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:44.257987 (+ 111us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:44.258337 (+ 350us) server_negotiation.cc:407] Connection header received
0811 03:05:44.258579 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:44.258586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:44.258657 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:44.258832 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:44.259824 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.260547 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.261796 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.262269 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.262412 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:44.273939 (+ 11527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:42333: BlockingRecv error: recv got EOF from 127.27.109.1:42333 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 03:05:44 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:44 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:44.648594 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:44.637689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:36387)
0811 03:05:44.638303 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:44.638320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:44.638444 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:44.638894 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:44.638898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:44.639376 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:44.639534 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:44.639543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.641045 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.641051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:44.642309 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:44.642325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.643237 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.643244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:44.643446 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:44.644062 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:44.644082 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:44.648195 (+ 4113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":364,"threads_started":1}
W20260811 03:05:44.648742 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:44.637807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:36387 (local address 127.27.109.62:34745)
0811 03:05:44.638055 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:44.638068 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:44.638415 (+ 347us) server_negotiation.cc:407] Connection header received
0811 03:05:44.638589 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:44.638594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:44.638663 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:44.638805 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:44.639693 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.640853 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.642487 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.643050 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.643208 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:44.648456 (+ 5248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:36387: BlockingRecv error: recv got EOF from 127.27.109.3:36387 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 11 03:05:44 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:44 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:44.831954 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:44.822737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:57349)
0811 03:05:44.823307 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:44.823326 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:44.823459 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:44.823819 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:44.823823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:44.824254 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:44.824428 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:44.824438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.825501 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.825506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:44.826442 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:44.826454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.827322 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.827327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:44.827461 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:44.828044 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:44.828061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:44.831657 (+ 3596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":353,"threads_started":1}
W20260811 03:05:44.832260 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:44.822896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:57349 (local address 127.27.109.62:34745)
0811 03:05:44.823214 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:44.823220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:44.823409 (+ 189us) server_negotiation.cc:407] Connection header received
0811 03:05:44.823574 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:44.823580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:44.823646 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:44.823730 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:44.824576 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.825327 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.826588 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:44.827170 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:44.827265 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:44.831899 (+ 4634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:57349: BlockingRecv error: recv got EOF from 127.27.109.2:57349 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 11 03:05:45 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:45 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:45.285802 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:45.275254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:41687)
0811 03:05:45.275978 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:45.276002 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:45.276163 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:45.276582 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:45.276587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:45.277141 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:45.277345 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:45.277356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.278487 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.278494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:45.279535 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:45.279549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.280391 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.280398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:45.280580 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:45.281246 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:45.281265 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:45.285444 (+ 4179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":314,"threads_started":1}
W20260811 03:05:45.286000 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:45.275328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:41687 (local address 127.27.109.62:34745)
0811 03:05:45.275538 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:45.275563 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:45.276156 (+ 593us) server_negotiation.cc:407] Connection header received
0811 03:05:45.276276 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:45.276281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:45.276369 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:45.276538 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:45.277507 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.278316 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.279717 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.280213 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.280366 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:45.285742 (+ 5376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:41687: BlockingRecv error: recv got EOF from 127.27.109.1:41687 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 11 03:05:45 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:45 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:45.660893 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:45.649760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:54021)
0811 03:05:45.650367 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:45.650421 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:45.650605 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:45.651060 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:45.651064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:45.651549 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:45.651726 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:45.651736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.652815 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.652820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:45.653787 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:45.653800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.654629 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.654635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:45.654867 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:45.655522 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:45.655542 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:45.660570 (+ 5028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":284,"threads_started":1}
W20260811 03:05:45.661296 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:45.650048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:54021 (local address 127.27.109.62:34745)
0811 03:05:45.650323 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:45.650330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:45.650519 (+ 189us) server_negotiation.cc:407] Connection header received
0811 03:05:45.650776 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:45.650783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:45.650854 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:45.651023 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:45.651899 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.652660 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.653982 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.654452 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.654600 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:45.660979 (+ 6379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:54021: BlockingRecv error: recv got EOF from 127.27.109.3:54021 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 11 03:05:45 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:45 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:45.843328 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:45.833137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:59569)
0811 03:05:45.833775 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:45.833794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:45.833934 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:45.834316 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:45.834320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:45.834814 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:45.835037 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:45.835051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.836170 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.836175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:45.837163 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:45.837176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.837987 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.837993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:45.838119 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:45.838879 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:45.838899 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:45.843043 (+ 4144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":294,"threads_started":1}
W20260811 03:05:45.843593 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:45.833214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:59569 (local address 127.27.109.62:34745)
0811 03:05:45.833493 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:45.833503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:45.833894 (+ 391us) server_negotiation.cc:407] Connection header received
0811 03:05:45.834063 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:45.834067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:45.834132 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:45.834220 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:45.835224 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.835995 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.837319 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:45.837809 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:45.837911 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:45.843276 (+ 5365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:59569: BlockingRecv error: recv got EOF from 127.27.109.2:59569 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 11 03:05:46 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:46 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:46.297437 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:46.286940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:33373)
0811 03:05:46.287518 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:46.287536 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:46.287663 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:46.288045 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:46.288049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:46.288525 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:46.288721 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:46.288732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.289928 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.289933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:46.290966 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:46.290981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.291846 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.291851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:46.292007 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:46.292711 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:46.292733 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:46.297023 (+ 4290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":269,"threads_started":1}
W20260811 03:05:46.297667 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:46.287138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:33373 (local address 127.27.109.62:34745)
0811 03:05:46.287376 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:46.287385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:46.287649 (+ 264us) server_negotiation.cc:407] Connection header received
0811 03:05:46.287757 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:46.287761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:46.287826 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:46.287916 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:46.288976 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.289742 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.291143 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.291626 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.291751 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:46.297396 (+ 5645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:33373: BlockingRecv error: recv got EOF from 127.27.109.1:33373 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 11 03:05:46 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:46 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:46.674033 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:46.662322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:59081)
0811 03:05:46.663086 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:46.663106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:46.663321 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:46.663897 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:46.663902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:46.664442 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:46.664710 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:46.664721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.666378 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.666385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:46.667623 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:46.667642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.668615 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.668620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:46.668758 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:46.669445 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:46.669469 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:46.673678 (+ 4209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":368,"threads_started":1}
W20260811 03:05:46.674404 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:46.662635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:59081 (local address 127.27.109.62:34745)
0811 03:05:46.662864 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:46.662871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:46.663315 (+ 444us) server_negotiation.cc:407] Connection header received
0811 03:05:46.663543 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:46.663549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:46.663658 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:46.663799 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:46.664925 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.666117 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.667823 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.668410 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.668531 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:46.674109 (+ 5578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:59081: BlockingRecv error: recv got EOF from 127.27.109.3:59081 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 11 03:05:46 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:46 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:46.856976 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:46.844649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:56373)
0811 03:05:46.845361 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:46.845389 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:46.845615 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:46.846023 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:46.846027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:46.846499 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:46.846772 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:46.846789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.848558 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.848562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:46.849842 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:46.849861 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.851030 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.851036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:46.851181 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:46.851969 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:46.851992 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:46.856638 (+ 4646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":337,"threads_started":1}
W20260811 03:05:46.857298 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:46.844944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:56373 (local address 127.27.109.62:34745)
0811 03:05:46.845310 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:46.845319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:46.845494 (+ 175us) server_negotiation.cc:407] Connection header received
0811 03:05:46.845779 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:46.845784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:46.845864 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:46.845981 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:46.846997 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.848351 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.850040 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:46.850780 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:46.850916 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:46.856956 (+ 6040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:56373: BlockingRecv error: recv got EOF from 127.27.109.2:56373 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Aug 11 03:05:47 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:47 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:47.310947 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:47.298826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:39659)
0811 03:05:47.299501 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:47.299520 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:47.299653 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:47.300174 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:47.300177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:47.300669 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:47.300836 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:47.300846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.302554 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.302559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:47.303673 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:47.303717 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.304766 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.304771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:47.304989 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:47.305923 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:47.305953 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:47.310588 (+ 4635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":357,"threads_started":1}
W20260811 03:05:47.311336 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:47.298898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:39659 (local address 127.27.109.62:34745)
0811 03:05:47.299241 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:47.299249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:47.299621 (+ 372us) server_negotiation.cc:407] Connection header received
0811 03:05:47.299817 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:47.299822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:47.299897 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:47.300076 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:47.301210 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.302331 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.304020 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.304541 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.304706 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:47.311080 (+ 6374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:39659: BlockingRecv error: recv got EOF from 127.27.109.1:39659 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 11 03:05:47 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:47 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:47.685645 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:47.675210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:46693)
0811 03:05:47.675829 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:47.675851 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:47.676035 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:47.676463 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:47.676466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:47.676894 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:47.677090 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:47.677099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.678338 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.678342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:47.679368 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:47.679381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.680315 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.680322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:47.680503 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:47.681248 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:47.681269 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:47.685320 (+ 4051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":305,"threads_started":1}
W20260811 03:05:47.685866 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:47.675296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:46693 (local address 127.27.109.62:34745)
0811 03:05:47.675609 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:47.675617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:47.675924 (+ 307us) server_negotiation.cc:407] Connection header received
0811 03:05:47.676219 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:47.676228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:47.676311 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:47.676419 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:47.677254 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.678179 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.679534 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.680061 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.680192 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:47.685632 (+ 5440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:46693: BlockingRecv error: recv got EOF from 127.27.109.3:46693 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 11 03:05:47 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:47 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:47.869864 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:47.858292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:54533)
0811 03:05:47.858964 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:47.858983 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:47.859120 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:47.859619 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:47.859623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:47.860120 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:47.860315 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:47.860326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.861564 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.861572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:47.862593 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:47.862608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.863633 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.863639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:47.863800 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:47.864593 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:47.864615 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:47.869492 (+ 4877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":391,"threads_started":1}
W20260811 03:05:47.870143 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:47.858370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:54533 (local address 127.27.109.62:34745)
0811 03:05:47.858626 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:47.858637 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:47.859106 (+ 469us) server_negotiation.cc:407] Connection header received
0811 03:05:47.859325 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:47.859329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:47.859422 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:47.859537 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:47.860494 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.861339 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.862833 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:47.863411 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:47.863524 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:47.869829 (+ 6305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:54533: BlockingRecv error: recv got EOF from 127.27.109.2:54533 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 11 03:05:48 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:48 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:48.323801 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:48.312394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:43321)
0811 03:05:48.313169 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:48.313194 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:48.313364 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:48.313788 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:48.313793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:48.314335 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:48.314588 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:48.314600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.315839 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.315856 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:48.317007 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:48.317021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.317892 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.317900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:48.318111 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:48.318921 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:48.318944 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:48.323358 (+ 4414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":355,"threads_started":1}
W20260811 03:05:48.323945 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:48.312480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:43321 (local address 127.27.109.62:34745)
0811 03:05:48.312759 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:48.312774 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:48.313276 (+ 502us) server_negotiation.cc:407] Connection header received
0811 03:05:48.313531 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:48.313543 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:48.313615 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:48.313739 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:48.314809 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.315658 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.317184 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.317694 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.317854 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:48.323605 (+ 5751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:43321: BlockingRecv error: recv got EOF from 127.27.109.1:43321 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 11 03:05:48 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:48 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:48.698271 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:48.687086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:47857)
0811 03:05:48.687678 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:48.687694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:48.687942 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:48.688335 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:48.688340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:48.688950 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:48.689154 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:48.689164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.690610 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.690614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:48.691659 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:48.691673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.692560 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.692565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:48.692702 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:48.693473 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:48.693493 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:48.697836 (+ 4343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":302,"threads_started":1}
W20260811 03:05:48.698539 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:48.687106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:47857 (local address 127.27.109.62:34745)
0811 03:05:48.687449 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:48.687458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:48.687804 (+ 346us) server_negotiation.cc:407] Connection header received
0811 03:05:48.688101 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:48.688114 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:48.688181 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:48.688294 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:48.689373 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.690463 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.691833 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.692366 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.692489 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:48.698227 (+ 5738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:47857: BlockingRecv error: recv got EOF from 127.27.109.3:47857 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 11 03:05:48 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:48 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:48.882040 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:48.871520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:42609)
0811 03:05:48.872257 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:48.872281 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:48.872443 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:48.873044 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:48.873050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:48.873661 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:48.873916 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:48.873928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.875245 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.875249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:48.876286 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:48.876300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.877100 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.877106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:48.877237 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:48.877953 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:48.877973 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:48.881776 (+ 3803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":444,"threads_started":1}
W20260811 03:05:48.882639 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:48.871575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:42609 (local address 127.27.109.62:34745)
0811 03:05:48.871782 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:48.871787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:48.872405 (+ 618us) server_negotiation.cc:407] Connection header received
0811 03:05:48.872657 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:48.872663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:48.872815 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:48.872966 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:48.874179 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.875069 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.876453 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:48.876924 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:48.877068 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:48.882224 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:42609: BlockingRecv error: recv got EOF from 127.27.109.2:42609 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 11 03:05:49 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:49 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:49.340173 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:49.325137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:45013)
0811 03:05:49.325746 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:49.325770 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:49.325936 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:49.326311 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:49.326317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:49.326940 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:49.327180 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:49.327193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.328365 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.328371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:49.330013 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:49.330034 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.331162 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.331169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:49.331405 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:49.335954 (+ 4549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:49.335985 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:49.339918 (+ 3933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":372,"threads_started":1}
W20260811 03:05:49.340646 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:49.325385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:45013 (local address 127.27.109.62:34745)
0811 03:05:49.325599 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:49.325606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:49.325901 (+ 295us) server_negotiation.cc:407] Connection header received
0811 03:05:49.326084 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:49.326088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:49.326157 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:49.326249 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:49.327375 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.328207 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.330243 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.330965 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.331094 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:49.340272 (+ 9178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:45013: BlockingRecv error: recv got EOF from 127.27.109.1:45013 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 11 03:05:49 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:49 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:49.709738 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:49.699440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:39685)
0811 03:05:49.699960 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:49.699979 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:49.700128 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:49.700502 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:49.700506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:49.700895 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:49.701183 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:49.701196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.702424 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.702428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:49.703522 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:49.703537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.704301 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.704306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:49.704429 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:49.705157 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:49.705177 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:49.709394 (+ 4217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":298,"threads_started":1}
W20260811 03:05:49.709921 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:49.699516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:39685 (local address 127.27.109.62:34745)
0811 03:05:49.699763 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:49.699770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:49.700091 (+ 321us) server_negotiation.cc:407] Connection header received
0811 03:05:49.700243 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:49.700247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:49.700323 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:49.700415 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:49.701351 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.702284 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.703671 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.704150 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.704255 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:49.709650 (+ 5395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:39685: BlockingRecv error: recv got EOF from 127.27.109.3:39685 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 11 03:05:49 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:49 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:49.893754 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:49.883181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:55409)
0811 03:05:49.883726 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:49.883743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:49.883882 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:49.884340 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:49.884345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:49.884859 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:49.885050 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:49.885059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.886417 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.886425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:49.887520 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:49.887535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.888416 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.888421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:49.888628 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:49.889302 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:49.889320 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:49.893397 (+ 4077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":278,"threads_started":1}
W20260811 03:05:49.893868 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:49.883313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:55409 (local address 127.27.109.62:34745)
0811 03:05:49.883568 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:49.883575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:49.883831 (+ 256us) server_negotiation.cc:407] Connection header received
0811 03:05:49.884089 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:49.884096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:49.884190 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:49.884300 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:49.885231 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.886267 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.887713 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:49.888199 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:49.888364 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:49.893600 (+ 5236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:55409: BlockingRecv error: recv got EOF from 127.27.109.2:55409 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 11 03:05:50 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:50 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:50.351903 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:50.341310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:54109)
0811 03:05:50.341941 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:50.341957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:50.342139 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:50.342526 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:50.342533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:50.343080 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:50.343266 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:50.343276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.344403 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.344408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:50.345399 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:50.345411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.346190 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.346196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:50.346390 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:50.347271 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:50.347290 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:50.351603 (+ 4313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":342,"threads_started":1}
W20260811 03:05:50.352010 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:50.341394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:54109 (local address 127.27.109.62:34745)
0811 03:05:50.341675 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:50.341684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:50.342071 (+ 387us) server_negotiation.cc:407] Connection header received
0811 03:05:50.342281 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:50.342293 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:50.342356 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:50.342474 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:50.343480 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.344220 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.345551 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.345994 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.346137 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:50.351816 (+ 5679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:54109: BlockingRecv error: recv got EOF from 127.27.109.1:54109 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 11 03:05:50 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:50 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:50.721724 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:50.710976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:37471)
0811 03:05:50.711533 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:50.711551 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:50.711734 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:50.712165 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:50.712169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:50.712684 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:50.712907 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:50.712917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.714146 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.714150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:50.715243 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:50.715256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.716084 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.716090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:50.716332 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:50.716968 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:50.716989 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:50.721409 (+ 4420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":298,"threads_started":1}
W20260811 03:05:50.721896 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:50.711035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:37471 (local address 127.27.109.62:34745)
0811 03:05:50.711302 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:50.711312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:50.711657 (+ 345us) server_negotiation.cc:407] Connection header received
0811 03:05:50.711887 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:50.711892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:50.711978 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:50.712134 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:50.713076 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.713953 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.715407 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.715886 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.716031 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:50.721648 (+ 5617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:37471: BlockingRecv error: recv got EOF from 127.27.109.3:37471 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 11 03:05:50 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:50 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:50.906176 6226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:50.895203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:45915)
0811 03:05:50.895826 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:50.895845 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:50.895980 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:50.896416 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:50.896419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:50.896908 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:50.897115 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:50.897127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.898404 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.898408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:50.899824 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:50.899839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.900722 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.900727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:50.900897 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:50.901616 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:50.901635 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:50.905771 (+ 4136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":356,"threads_started":1}
W20260811 03:05:50.906440 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:50.895287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:45915 (local address 127.27.109.62:34745)
0811 03:05:50.895518 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:50.895523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:50.895942 (+ 419us) server_negotiation.cc:407] Connection header received
0811 03:05:50.896140 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:50.896144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:50.896222 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:50.896320 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:50.897290 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.898227 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.899973 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:50.900540 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:50.900646 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:50.906137 (+ 5491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:45915: BlockingRecv error: recv got EOF from 127.27.109.2:45915 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 11 03:05:51 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:51 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:51.363696 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:51.353065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:49801)
0811 03:05:51.353696 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:51.353732 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:51.353977 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:51.354373 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:51.354378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:51.354969 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:51.355144 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:51.355154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.356243 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.356251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:51.357207 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:51.357219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.358009 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.358015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:51.358210 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:51.359115 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:51.359135 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:51.363307 (+ 4172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":372,"threads_started":1}
W20260811 03:05:51.363792 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:51.353266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:49801 (local address 127.27.109.62:34745)
0811 03:05:51.353568 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:51.353575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:51.353829 (+ 254us) server_negotiation.cc:407] Connection header received
0811 03:05:51.354123 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:51.354129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:51.354218 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:51.354335 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:51.355303 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.356088 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.357364 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.357850 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.357991 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:51.363527 (+ 5536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:49801: BlockingRecv error: recv got EOF from 127.27.109.1:49801 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 11 03:05:51 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:51 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:51.734776 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:51.723078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:57889)
0811 03:05:51.723757 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:51.723784 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:51.723948 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:51.724338 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:51.724343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:51.724804 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:51.725002 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:51.725012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.726294 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.726300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:51.727967 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:51.727985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.729337 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.729344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:51.729507 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:51.730240 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:51.730268 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:51.734438 (+ 4170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"spinlock_wait_cycles":36096,"thread_start_us":404,"threads_started":1}
W20260811 03:05:51.735085 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:51.723213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:57889 (local address 127.27.109.62:34745)
0811 03:05:51.723495 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:51.723503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:51.723874 (+ 371us) server_negotiation.cc:407] Connection header received
0811 03:05:51.724091 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:51.724098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:51.724179 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:51.724289 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:51.725162 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.726134 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.728174 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.728907 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.729047 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:51.734738 (+ 5691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:57889: BlockingRecv error: recv got EOF from 127.27.109.3:57889 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 11 03:05:51 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:51 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:51.917730 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:51.907430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:52297)
0811 03:05:51.908011 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:51.908029 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:51.908202 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:51.908631 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:51.908637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:51.909117 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:51.909270 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:51.909279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.910432 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.910437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:51.911458 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:51.911471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.912359 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.912366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:51.912555 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:51.913303 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:51.913322 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:51.917466 (+ 4144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":287,"threads_started":1}
W20260811 03:05:51.917961 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:51.907703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:52297 (local address 127.27.109.62:34745)
0811 03:05:51.907950 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:51.907957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:51.908116 (+ 159us) server_negotiation.cc:407] Connection header received
0811 03:05:51.908382 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:51.908387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:51.908469 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:51.908588 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:51.909445 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.910260 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.911615 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:51.912165 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:51.912309 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:51.917698 (+ 5389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:52297: BlockingRecv error: recv got EOF from 127.27.109.2:52297 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 11 03:05:52 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:52 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.1@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:52.375769 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:52.365239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.1:45525)
0811 03:05:52.365949 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:52.365965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:52.366150 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:52.366515 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:52.366520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:52.367079 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:52.367282 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:52.367293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.368362 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.368368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:52.369411 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:52.369425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.370202 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.370208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:52.370401 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:52.371096 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:52.371116 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:52.375497 (+ 4381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":396,"threads_started":1}
W20260811 03:05:52.376013 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:52.365460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:45525 (local address 127.27.109.62:34745)
0811 03:05:52.365755 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:52.365763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:52.366074 (+ 311us) server_negotiation.cc:407] Connection header received
0811 03:05:52.366290 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:52.366302 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:52.366369 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:52.366481 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:52.367465 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.368193 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.369573 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.370019 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.370172 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:52.375756 (+ 5584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.1:45525: BlockingRecv error: recv got EOF from 127.27.109.1:45525 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 11 03:05:52 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:52 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.3@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:52.747397 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:52.736228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.3:53755)
0811 03:05:52.736821 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:52.736838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:52.736970 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:52.737469 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:52.737473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:52.738034 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:52.738206 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:52.738216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.739445 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.739451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:52.740576 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:52.740592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.741473 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.741478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:52.741635 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:52.742439 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:52.742462 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:52.747154 (+ 4692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":341,"threads_started":1}
W20260811 03:05:52.747705 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:52.736280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:53755 (local address 127.27.109.62:34745)
0811 03:05:52.736629 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:52.736633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:52.736927 (+ 294us) server_negotiation.cc:407] Connection header received
0811 03:05:52.737148 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:52.737153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:52.737250 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:52.737357 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:52.738409 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.739252 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.740794 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.741271 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.741388 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:52.747401 (+ 6013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.3:53755: BlockingRecv error: recv got EOF from 127.27.109.3:53755 (error 108)
Metrics: {"server-negotiator.queue_time_us":226}
Aug 11 03:05:52 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 03:05:52 dist-test-slave-8tdl krb5kdc[5049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.109.2@KRBTEST.COM for oryx/127.27.109.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 03:05:52.930979 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:52.919426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34745 (local address 127.27.109.2:48953)
0811 03:05:52.920063 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:52.920083 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:52.920220 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:52.920593 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:52.920597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:52.921111 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:52.921315 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:52.921325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.922659 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.922714 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:52.923796 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:52.923812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.924686 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.924691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:52.924854 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:52.925672 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:05:52.925691 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:52.930612 (+ 4921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.109.62:34745: Server oryx/127.27.109.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":307,"threads_started":1}
W20260811 03:05:52.931294 5874 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:52.919669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:48953 (local address 127.27.109.62:34745)
0811 03:05:52.919933 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:52.919940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:52.920203 (+ 263us) server_negotiation.cc:407] Connection header received
0811 03:05:52.920344 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:52.920348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:52.920435 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:52.920536 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:52.921489 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.922382 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.923964 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:52.924510 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:52.924623 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:52.930997 (+ 6374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.109.2:48953: BlockingRecv error: recv got EOF from 127.27.109.2:48953 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 03:05:53.286274 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5709
I20260811 03:05:53.302000 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:53.287800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.27.109.1:35033)
0811 03:05:53.288459 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:53.288467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:53.288526 (+ 59us) server_negotiation.cc:407] Connection header received
0811 03:05:53.288625 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:53.288637 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:53.288936 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:53.289138 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:53.290940 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:53.292064 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:53.293394 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:53.294034 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:53.294188 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:53.297163 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:53.297176 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:53.297194 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:53.297263 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:53.301306 (+ 4043us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.109.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 03:05:53.301745 (+ 439us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38938: Request ticket server kudu/127.27.109.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":331,"threads_started":1}
W20260811 03:05:53.302155 6235 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38938: Request ticket server kudu/127.27.109.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260811 03:05:53.302302 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5709: Not authorized: Client connection negotiation failed: client connection to 127.27.109.1:35033: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.109.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 03:05:53.302546 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 5709
I20260811 03:05:53.325750 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5876
I20260811 03:05:53.346024 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:53.327073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.27.109.2:35395)
0811 03:05:53.330925 (+ 3852us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:53.330937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:53.330980 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:05:53.331071 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:53.331081 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:53.331434 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:53.331663 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:53.333381 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:53.334533 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:53.336071 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:53.337901 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:53.338229 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:53.341949 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:53.341962 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:53.341980 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:53.342063 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:53.345339 (+ 3276us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.109.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 03:05:53.345736 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43062: Request ticket server kudu/127.27.109.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":3568,"thread_start_us":3432,"threads_started":1}
W20260811 03:05:53.346199 6236 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43062: Request ticket server kudu/127.27.109.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260811 03:05:53.346276 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5876: Not authorized: Client connection negotiation failed: client connection to 127.27.109.2:35395: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.109.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 03:05:53.346506 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 5876
I20260811 03:05:53.370114 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 6033
W20260811 03:05:53.385807 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 6033: Not authorized: Client connection negotiation failed: client connection to 127.27.109.3:45353: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.109.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 03:05:53.385862 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:53.371397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.27.109.3:45353)
0811 03:05:53.372022 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:53.372029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:53.372063 (+ 34us) server_negotiation.cc:407] Connection header received
0811 03:05:53.372131 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:53.372138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:53.372390 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:53.372559 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:53.374659 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:53.375898 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:53.377304 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:53.378032 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:53.378169 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:53.381425 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:53.381436 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:53.381451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:05:53.381511 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:53.384871 (+ 3360us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.109.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 03:05:53.385417 (+ 546us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57832: Request ticket server kudu/127.27.109.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":307,"threads_started":1}
I20260811 03:05:53.385960 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 6033
W20260811 03:05:53.385980 6237 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57832: Request ticket server kudu/127.27.109.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 03:05:53.408087 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 5633
I20260811 03:05:53.409569 5701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:53.535620 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 5633
2026-08-11T03:05:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22374 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-11T03:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:53Z Disabled control of system clock
I20260811 03:05:53.638402 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38741
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 03:05:54.012018 6242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:54.012392 6242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:54.012537 6242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:54.020928 6242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:54.021090 6242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:54.021173 6242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:54.021276 6242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:54.021346 6242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:54.031391 6242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:54.032887 6242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:54.034924 6242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:54.047936 6250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:54.047936 6247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:54.047801 6248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:54.049086 6242 server_base.cc:1034] running on GCE node
I20260811 03:05:54.050119 6242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:54.051880 6242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:54.053184 6242 hybrid_clock.cc:648] HybridClock initialized: now 1786417554053103 us; error 61 us; skew 500 ppm
I20260811 03:05:54.053629 6242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:54.056260 6242 webserver.cc:460] Webserver started at http://127.27.109.62:37323/ using document root <none> and password file <none>
I20260811 03:05:54.057063 6242 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:54.057211 6242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:54.057559 6242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:54.060753 6242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "da3d5e0f63624cb4bb7cf1af8090777c"
format_stamp: "Formatted at 2026-08-11 03:05:54 on dist-test-slave-8tdl"
server_key: "11827a09725fc79bdfb1826913a72fad"
server_key_iv: "1d94b3658a942f8551706672caf968c4"
server_key_version: "encryptionkey@0"
I20260811 03:05:54.061618 6242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "da3d5e0f63624cb4bb7cf1af8090777c"
format_stamp: "Formatted at 2026-08-11 03:05:54 on dist-test-slave-8tdl"
server_key: "11827a09725fc79bdfb1826913a72fad"
server_key_iv: "1d94b3658a942f8551706672caf968c4"
server_key_version: "encryptionkey@0"
I20260811 03:05:54.067519 6242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 03:05:54.071725 6256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:54.073352 6242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:05:54.073575 6242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "da3d5e0f63624cb4bb7cf1af8090777c"
format_stamp: "Formatted at 2026-08-11 03:05:54 on dist-test-slave-8tdl"
server_key: "11827a09725fc79bdfb1826913a72fad"
server_key_iv: "1d94b3658a942f8551706672caf968c4"
server_key_version: "encryptionkey@0"
I20260811 03:05:54.073841 6242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:54.095072 6242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:54.096058 6242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:54.096386 6242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:54.116791 6242 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:38741
I20260811 03:05:54.116829 6319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:38741 every 8 connection(s)
I20260811 03:05:54.118314 6242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:54.120105 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 6242
I20260811 03:05:54.120362 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:54.121291 28084 external_mini_cluster.cc:1428] Setting key 3ba850235875edb1f59ba843398d0587
I20260811 03:05:54.123847 6320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:54.137389 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:54.123742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.27.109.62:38741)
0811 03:05:54.124962 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:54.124974 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:54.125022 (+ 48us) server_negotiation.cc:407] Connection header received
0811 03:05:54.127020 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:54.127046 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:54.127594 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:54.127985 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:54.128752 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:54.130317 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:54.132329 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:54.133065 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:54.133234 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:54.134775 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:54.134785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:54.134805 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:54.135485 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:54.135588 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:54.135611 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:54.135705 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:54.135976 (+ 271us) server_negotiation.cc:299] Negotiation successful
0811 03:05:54.136359 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":547,"threads_started":1}
I20260811 03:05:54.138262 6320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c: Bootstrap starting.
I20260811 03:05:54.141840 6320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:54.143232 6320 log.cc:824] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:54.146512 6320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c: No bootstrap required, opened a new log
I20260811 03:05:54.154630 6320 raft_consensus.cc:348] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3d5e0f63624cb4bb7cf1af8090777c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38741 } }
I20260811 03:05:54.155143 6320 raft_consensus.cc:374] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:54.155242 6320 raft_consensus.cc:729] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3d5e0f63624cb4bb7cf1af8090777c, State: Initialized, Role: FOLLOWER
I20260811 03:05:54.155858 6320 consensus_queue.cc:260] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3d5e0f63624cb4bb7cf1af8090777c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38741 } }
I20260811 03:05:54.156075 6320 raft_consensus.cc:388] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:54.156168 6320 raft_consensus.cc:482] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:54.156306 6320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:54.159060 6320 raft_consensus.cc:504] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3d5e0f63624cb4bb7cf1af8090777c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38741 } }
I20260811 03:05:54.159574 6320 leader_election.cc:302] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da3d5e0f63624cb4bb7cf1af8090777c; no voters:
I20260811 03:05:54.160199 6320 leader_election.cc:288] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:54.160413 6325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:05:54.161255 6325 raft_consensus.cc:686] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [term 1 LEADER]: Becoming Leader. State: Replica: da3d5e0f63624cb4bb7cf1af8090777c, State: Running, Role: LEADER
I20260811 03:05:54.161862 6325 consensus_queue.cc:237] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3d5e0f63624cb4bb7cf1af8090777c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38741 } }
I20260811 03:05:54.162743 6320 sys_catalog.cc:564] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:54.165285 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da3d5e0f63624cb4bb7cf1af8090777c. Latest consensus state: current_term: 1 leader_uuid: "da3d5e0f63624cb4bb7cf1af8090777c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3d5e0f63624cb4bb7cf1af8090777c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38741 } } }
I20260811 03:05:54.165706 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:54.165395 6326 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da3d5e0f63624cb4bb7cf1af8090777c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3d5e0f63624cb4bb7cf1af8090777c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38741 } } }
I20260811 03:05:54.165840 6326 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:54.168006 6332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:54.175935 6332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:54.186117 6332 catalog_manager.cc:1347] Generated new cluster ID: 0e56ca10b6904298a80129bb53147c91
I20260811 03:05:54.186276 6332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:54.207055 6332 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:54.208540 6332 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:54.220409 6332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da3d5e0f63624cb4bb7cf1af8090777c: Generated new TSK 0
I20260811 03:05:54.221388 6332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:05:54.235733 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38741
--builtin_ntp_servers=127.27.109.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260811 03:05:54.647202 6345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:54.647573 6345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:54.647732 6345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:54.655927 6345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:54.656107 6345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:54.656294 6345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:05:54.666261 6345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:54.667766 6345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:54.669700 6345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:54.684506 6350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:54.684551 6351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:54.684592 6353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:54.685508 6345 server_base.cc:1034] running on GCE node
I20260811 03:05:54.686144 6345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:54.687273 6345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:54.688529 6345 hybrid_clock.cc:648] HybridClock initialized: now 1786417554688472 us; error 38 us; skew 500 ppm
I20260811 03:05:54.688951 6345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:54.694679 6345 webserver.cc:460] Webserver started at http://127.27.109.1:44035/ using document root <none> and password file <none>
I20260811 03:05:54.695583 6345 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:54.695683 6345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:54.696137 6345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:54.699344 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "ddee705042f44a4c96446dfddd7be9b5"
format_stamp: "Formatted at 2026-08-11 03:05:54 on dist-test-slave-8tdl"
server_key: "41995bed67cf5a64255dd66c3cedb20f"
server_key_iv: "12476431b348158bf37598f2e6eb32d3"
server_key_version: "encryptionkey@0"
I20260811 03:05:54.700242 6345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "ddee705042f44a4c96446dfddd7be9b5"
format_stamp: "Formatted at 2026-08-11 03:05:54 on dist-test-slave-8tdl"
server_key: "41995bed67cf5a64255dd66c3cedb20f"
server_key_iv: "12476431b348158bf37598f2e6eb32d3"
server_key_version: "encryptionkey@0"
I20260811 03:05:54.706493 6345 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:05:54.710263 6359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:54.711936 6345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:54.712144 6345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "ddee705042f44a4c96446dfddd7be9b5"
format_stamp: "Formatted at 2026-08-11 03:05:54 on dist-test-slave-8tdl"
server_key: "41995bed67cf5a64255dd66c3cedb20f"
server_key_iv: "12476431b348158bf37598f2e6eb32d3"
server_key_version: "encryptionkey@0"
I20260811 03:05:54.712404 6345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:54.732321 6345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:54.733274 6345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:54.733625 6345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:54.734745 6345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:54.736408 6345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:54.736544 6345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:54.736691 6345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:54.736784 6345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:54.773233 6345 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:43313
I20260811 03:05:54.773301 6486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:43313 every 8 connection(s)
I20260811 03:05:54.774791 6345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:05:54.785225 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 6345
I20260811 03:05:54.785533 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:05:54.786264 28084 external_mini_cluster.cc:1428] Setting key 6bb371c74de5704e0f77fc4616c79825
I20260811 03:05:54.789623 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38741
--builtin_ntp_servers=127.27.109.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 03:05:54.790630 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:54.777769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:40277 (local address 127.27.109.62:38741)
0811 03:05:54.778393 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:54.778399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:54.779908 (+ 1509us) server_negotiation.cc:407] Connection header received
0811 03:05:54.781999 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:54.782005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:54.782121 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:54.782259 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:54.783814 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:54.784737 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:54.786867 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:54.787489 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:54.787648 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:54.789448 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:54.789453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:54.789457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:54.789477 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:54.789547 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:54.789562 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:54.789684 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:54.790031 (+ 347us) server_negotiation.cc:299] Negotiation successful
0811 03:05:54.790174 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":380,"threads_started":1}
I20260811 03:05:54.791678 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:54.778524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38741 (local address 127.27.109.1:40277)
0811 03:05:54.779694 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:54.779737 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:54.781658 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:54.782553 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:54.782563 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:54.782737 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:54.783610 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:54.783634 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:54.784969 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:54.784975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:54.786604 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:54.786626 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:54.787758 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:54.787765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:54.788046 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:54.788940 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:54.788943 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:54.788992 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:54.789177 (+ 185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:54.789686 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:54.789702 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:54.789713 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 03:05:54.789964 (+ 251us) client_negotiation.cc:240] Negotiation successful
0811 03:05:54.790339 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":377,"threads_started":1}
I20260811 03:05:54.793525 6487 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38741
I20260811 03:05:54.793874 6487 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:54.794970 6487 heartbeater.cc:511] Master 127.27.109.62:38741 requested a full tablet report, sending...
I20260811 03:05:54.797564 6276 ts_manager.cc:194] Registered new tserver with Master: ddee705042f44a4c96446dfddd7be9b5 (127.27.109.1:43313)
I20260811 03:05:54.799888 6276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.1:40277
W20260811 03:05:55.186921 6491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:55.187271 6491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:55.187408 6491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:55.195062 6491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:55.195279 6491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:55.195410 6491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:55.205081 6491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:55.206530 6491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:55.208518 6491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:55.223770 6500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:55.223770 6497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:55.223971 6498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:55.225776 6491 server_base.cc:1034] running on GCE node
I20260811 03:05:55.226642 6491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:55.227934 6491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:55.229262 6491 hybrid_clock.cc:648] HybridClock initialized: now 1786417555229204 us; error 57 us; skew 500 ppm
I20260811 03:05:55.229701 6491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:55.232865 6491 webserver.cc:460] Webserver started at http://127.27.109.2:41189/ using document root <none> and password file <none>
I20260811 03:05:55.233740 6491 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:55.233918 6491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:55.234318 6491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:55.237982 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "e124a1b771b94230a89eae93f37e9fb4"
format_stamp: "Formatted at 2026-08-11 03:05:55 on dist-test-slave-8tdl"
server_key: "cc63326ba01ea12e39233428a6f9124b"
server_key_iv: "98afa38af501aba542a5e23150d67879"
server_key_version: "encryptionkey@0"
I20260811 03:05:55.239171 6491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "e124a1b771b94230a89eae93f37e9fb4"
format_stamp: "Formatted at 2026-08-11 03:05:55 on dist-test-slave-8tdl"
server_key: "cc63326ba01ea12e39233428a6f9124b"
server_key_iv: "98afa38af501aba542a5e23150d67879"
server_key_version: "encryptionkey@0"
I20260811 03:05:55.245790 6491 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:05:55.249972 6506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:55.251680 6491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:05:55.251911 6491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "e124a1b771b94230a89eae93f37e9fb4"
format_stamp: "Formatted at 2026-08-11 03:05:55 on dist-test-slave-8tdl"
server_key: "cc63326ba01ea12e39233428a6f9124b"
server_key_iv: "98afa38af501aba542a5e23150d67879"
server_key_version: "encryptionkey@0"
I20260811 03:05:55.252175 6491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:55.271440 6491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:55.272433 6491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:55.272789 6491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:55.274019 6491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:55.275759 6491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:55.275888 6491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:55.276031 6491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:55.276124 6491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:55.314522 6491 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:39987
I20260811 03:05:55.314591 6635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:39987 every 8 connection(s)
I20260811 03:05:55.316115 6491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:55.324771 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 6491
I20260811 03:05:55.325011 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:05:55.325490 28084 external_mini_cluster.cc:1428] Setting key e64918418a348b0413091e028cd33861
I20260811 03:05:55.329036 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38741
--builtin_ntp_servers=127.27.109.20:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 03:05:55.332599 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:55.319226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:57771 (local address 127.27.109.62:38741)
0811 03:05:55.319830 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:55.319836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:55.321444 (+ 1608us) server_negotiation.cc:407] Connection header received
0811 03:05:55.323629 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:55.323635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:55.323743 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:55.323877 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:55.325325 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.326228 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.328778 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.329318 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.329447 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:55.331619 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:55.331625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:55.331629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:55.331653 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:55.331701 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:55.331715 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:55.331800 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:55.332163 (+ 363us) server_negotiation.cc:299] Negotiation successful
0811 03:05:55.332286 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":340,"threads_started":1}
I20260811 03:05:55.334313 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:55.319682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38741 (local address 127.27.109.2:57771)
0811 03:05:55.321026 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:55.321269 (+ 243us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:55.323259 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:55.324188 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:55.324198 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:55.324308 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:55.325091 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:55.325128 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.326460 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.326466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:55.328475 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:55.328504 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.329697 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.329713 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:55.330118 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:55.331120 (+ 1002us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:55.331127 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:55.331195 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:55.331334 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:55.331838 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:55.331853 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:55.331872 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 03:05:55.332088 (+ 216us) client_negotiation.cc:240] Negotiation successful
0811 03:05:55.332638 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":379,"threads_started":1}
I20260811 03:05:55.336383 6636 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38741
I20260811 03:05:55.336855 6636 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:55.337831 6636 heartbeater.cc:511] Master 127.27.109.62:38741 requested a full tablet report, sending...
I20260811 03:05:55.339828 6276 ts_manager.cc:194] Registered new tserver with Master: e124a1b771b94230a89eae93f37e9fb4 (127.27.109.2:39987)
I20260811 03:05:55.341138 6276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.2:57771
W20260811 03:05:55.726116 6640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:55.726634 6640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:55.726822 6640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:55.737007 6640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:55.737144 6640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:55.737270 6640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:55.747495 6640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:55.748908 6640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:55.751065 6640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:55.766088 6645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:55.766129 6647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:55.766184 6649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:55.767685 6640 server_base.cc:1034] running on GCE node
I20260811 03:05:55.768785 6640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:55.770040 6640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:55.771287 6640 hybrid_clock.cc:648] HybridClock initialized: now 1786417555771235 us; error 52 us; skew 500 ppm
I20260811 03:05:55.771708 6640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:55.775457 6640 webserver.cc:460] Webserver started at http://127.27.109.3:44227/ using document root <none> and password file <none>
I20260811 03:05:55.776563 6640 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:55.776717 6640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:55.777186 6640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:55.781523 6640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "a7a748c6a180416c9de204ac94c81737"
format_stamp: "Formatted at 2026-08-11 03:05:55 on dist-test-slave-8tdl"
server_key: "f35659e788212797e6539514baf2062c"
server_key_iv: "1702cabd98599e1110c6f37f19af9c82"
server_key_version: "encryptionkey@0"
I20260811 03:05:55.782461 6640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "a7a748c6a180416c9de204ac94c81737"
format_stamp: "Formatted at 2026-08-11 03:05:55 on dist-test-slave-8tdl"
server_key: "f35659e788212797e6539514baf2062c"
server_key_iv: "1702cabd98599e1110c6f37f19af9c82"
server_key_version: "encryptionkey@0"
I20260811 03:05:55.789196 6640 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:05:55.793637 6655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:55.795513 6640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260811 03:05:55.795686 6640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "a7a748c6a180416c9de204ac94c81737"
format_stamp: "Formatted at 2026-08-11 03:05:55 on dist-test-slave-8tdl"
server_key: "f35659e788212797e6539514baf2062c"
server_key_iv: "1702cabd98599e1110c6f37f19af9c82"
server_key_version: "encryptionkey@0"
I20260811 03:05:55.796075 6640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:55.803500 6487 heartbeater.cc:503] Master 127.27.109.62:38741 was elected leader, sending a full tablet report...
I20260811 03:05:55.825932 6640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:55.827087 6640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:55.827456 6640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:55.828590 6640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:55.830324 6640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:55.830466 6640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:55.830626 6640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:55.830756 6640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260811 03:05:55.870795 6640 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:37287
I20260811 03:05:55.870863 6782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:37287 every 8 connection(s)
I20260811 03:05:55.872339 6640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:55.882740 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 6640
I20260811 03:05:55.882987 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:05:55.883589 28084 external_mini_cluster.cc:1428] Setting key d97c73cda20b0dbdcc79bf3e90d82c06
I20260811 03:05:55.888327 6785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:55.875460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:44173 (local address 127.27.109.62:38741)
0811 03:05:55.875965 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:55.875972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:55.877482 (+ 1510us) server_negotiation.cc:407] Connection header received
0811 03:05:55.879836 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:55.879840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:55.879921 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:55.880037 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:55.881427 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.882222 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.884774 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.885352 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.885455 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:55.887351 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:55.887359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:55.887365 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:55.887388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:55.887446 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:55.887467 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:55.887591 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:55.887912 (+ 321us) server_negotiation.cc:299] Negotiation successful
0811 03:05:55.888012 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":236,"threads_started":1}
I20260811 03:05:55.889479 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:55.876160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38741 (local address 127.27.109.3:44173)
0811 03:05:55.877298 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:55.877338 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:55.879512 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:55.880301 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:55.880311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:55.880456 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:55.881208 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:55.881230 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.882463 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.882470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:55.884615 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:05:55.884630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.885641 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.885650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:05:55.885981 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:55.886880 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:55.886883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:55.886939 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:55.887100 (+ 161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:55.887642 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:55.887655 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:55.887666 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 03:05:55.887854 (+ 188us) client_negotiation.cc:240] Negotiation successful
0811 03:05:55.888226 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":436,"threads_started":1}
I20260811 03:05:55.891196 6783 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38741
I20260811 03:05:55.891552 6783 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:55.892493 6783 heartbeater.cc:511] Master 127.27.109.62:38741 requested a full tablet report, sending...
I20260811 03:05:55.894227 6276 ts_manager.cc:194] Registered new tserver with Master: a7a748c6a180416c9de204ac94c81737 (127.27.109.3:37287)
I20260811 03:05:55.895232 6276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.3:44173
I20260811 03:05:55.900478 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 03:05:55.907670 6785 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:55.905862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.27.109.62:38741)
0811 03:05:55.906165 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:55.906171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:55.906597 (+ 426us) server_negotiation.cc:407] Connection header received
0811 03:05:55.906816 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:55.906821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:55.906894 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:55.907014 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:55.907436 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47984: BlockingRecv error: recv got EOF from 127.0.0.1:47984 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
I20260811 03:05:55.908457 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 6345
I20260811 03:05:55.917986 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:55.909652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.27.109.1:43313)
0811 03:05:55.910331 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:55.910339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:55.910368 (+ 29us) server_negotiation.cc:407] Connection header received
0811 03:05:55.910439 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:55.910452 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:55.910744 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:55.910930 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:55.911485 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.912687 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.914116 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:55.914852 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:55.914964 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:55.916270 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:55.916279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:55.916293 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:55.917007 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:55.917074 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:55.917097 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:55.917211 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:55.917516 (+ 305us) server_negotiation.cc:299] Negotiation successful
0811 03:05:55.917656 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":363,"threads_started":1}
I20260811 03:05:55.919335 6482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:56.005645 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 6345
I20260811 03:05:56.028519 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 6491
I20260811 03:05:56.037313 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:56.029534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.27.109.2:39987)
0811 03:05:56.030116 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:56.030124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:56.030151 (+ 27us) server_negotiation.cc:407] Connection header received
0811 03:05:56.030221 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:56.030228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:56.030455 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:56.030625 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:56.031180 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:56.032313 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:56.033706 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:56.034381 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:56.034491 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:56.035740 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:56.035748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:56.035763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:56.036386 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:56.036438 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:56.036459 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:56.036563 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:56.036716 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 03:05:56.036890 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":252,"threads_started":1}
I20260811 03:05:56.038640 6631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:56.124234 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 6491
I20260811 03:05:56.144418 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 6640
I20260811 03:05:56.153211 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:56.145376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.27.109.3:37287)
0811 03:05:56.146083 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:56.146091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:56.146124 (+ 33us) server_negotiation.cc:407] Connection header received
0811 03:05:56.146186 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:56.146192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:56.146414 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:56.146554 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:56.147136 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:56.148212 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:56.149541 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:05:56.150184 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:05:56.150289 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:05:56.151530 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:56.151539 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:56.151553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:56.152183 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:56.152234 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:56.152255 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:56.152349 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:56.152643 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 03:05:56.152784 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":372,"threads_started":1}
I20260811 03:05:56.154670 6778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:56.242352 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 6640
I20260811 03:05:56.263728 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 6242
I20260811 03:05:56.265193 6314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:56.378508 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 6242
2026-08-11T03:05:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2837 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-11T03:05:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:05:56Z Disabled control of system clock
I20260811 03:05:56.477595 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39413
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:39413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 03:05:56.846235 6803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:56.846576 6803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:56.846666 6803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:56.854257 6803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:05:56.854395 6803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:56.854434 6803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:56.854486 6803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:05:56.854532 6803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:05:56.864697 6803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:39413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:39413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:56.866112 6803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:56.868093 6803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:56.880746 6809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:56.880746 6812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:56.881331 6810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:56.881753 6803 server_base.cc:1034] running on GCE node
I20260811 03:05:56.882850 6803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:56.884443 6803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:56.885715 6803 hybrid_clock.cc:648] HybridClock initialized: now 1786417556885664 us; error 50 us; skew 500 ppm
I20260811 03:05:56.886129 6803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:56.889001 6803 webserver.cc:460] Webserver started at http://127.27.109.62:37257/ using document root <none> and password file <none>
I20260811 03:05:56.889886 6803 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:56.890048 6803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:56.890450 6803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:56.894333 6803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "b87d2d2f012d448faa5f212a3bd33b0a"
format_stamp: "Formatted at 2026-08-11 03:05:56 on dist-test-slave-8tdl"
server_key: "1400e4f8c60e0950213b55a59af8a392"
server_key_iv: "1f3233e772bd4f33c9f1f297b6e0628e"
server_key_version: "encryptionkey@0"
I20260811 03:05:56.895491 6803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "b87d2d2f012d448faa5f212a3bd33b0a"
format_stamp: "Formatted at 2026-08-11 03:05:56 on dist-test-slave-8tdl"
server_key: "1400e4f8c60e0950213b55a59af8a392"
server_key_iv: "1f3233e772bd4f33c9f1f297b6e0628e"
server_key_version: "encryptionkey@0"
I20260811 03:05:56.901825 6803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260811 03:05:56.906333 6819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:56.908193 6803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:05:56.908435 6803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "b87d2d2f012d448faa5f212a3bd33b0a"
format_stamp: "Formatted at 2026-08-11 03:05:56 on dist-test-slave-8tdl"
server_key: "1400e4f8c60e0950213b55a59af8a392"
server_key_iv: "1f3233e772bd4f33c9f1f297b6e0628e"
server_key_version: "encryptionkey@0"
I20260811 03:05:56.908756 6803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:56.915643 6803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:56.916636 6803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:56.917011 6803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:56.938926 6803 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:39413
I20260811 03:05:56.938967 6875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:39413 every 8 connection(s)
I20260811 03:05:56.940511 6803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:05:56.940943 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 6803
I20260811 03:05:56.941277 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:05:56.942241 28084 external_mini_cluster.cc:1428] Setting key 3e2aced2ec24237a0b117f8fb0d289b8
I20260811 03:05:56.945788 6876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:05:56.952430 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:56.944814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.27.109.62:39413)
0811 03:05:56.946198 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:56.946210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:56.946256 (+ 46us) server_negotiation.cc:407] Connection header received
0811 03:05:56.948246 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:56.948272 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:56.948736 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:56.949227 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:56.950448 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:56.950455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:56.950473 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:56.950492 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:56.950584 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:56.950605 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:56.950890 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:56.951049 (+ 159us) server_negotiation.cc:299] Negotiation successful
0811 03:05:56.951348 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":584,"threads_started":1}
I20260811 03:05:56.958995 6876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a: Bootstrap starting.
I20260811 03:05:56.962440 6876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a: Neither blocks nor log segments found. Creating new log.
I20260811 03:05:56.963624 6876 log.cc:824] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a: Log is configured to *not* fsync() on all Append() calls
I20260811 03:05:56.966598 6876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a: No bootstrap required, opened a new log
I20260811 03:05:56.974655 6876 raft_consensus.cc:348] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39413 } }
I20260811 03:05:56.975170 6876 raft_consensus.cc:374] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:05:56.975299 6876 raft_consensus.cc:729] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87d2d2f012d448faa5f212a3bd33b0a, State: Initialized, Role: FOLLOWER
I20260811 03:05:56.976177 6876 consensus_queue.cc:260] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39413 } }
I20260811 03:05:56.976512 6881 raft_consensus.cc:482] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:05:56.976814 6881 raft_consensus.cc:504] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39413 } }
I20260811 03:05:56.977526 6881 leader_election.cc:302] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87d2d2f012d448faa5f212a3bd33b0a; no voters:
I20260811 03:05:56.978328 6881 leader_election.cc:288] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260811 03:05:56.976455 6876 raft_consensus.cc:388] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:05:56.978629 6882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:05:56.978837 6882 raft_consensus.cc:482] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:05:56.978981 6882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:05:56.981299 6882 raft_consensus.cc:504] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39413 } }
I20260811 03:05:56.981642 6882 leader_election.cc:302] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87d2d2f012d448faa5f212a3bd33b0a; no voters:
I20260811 03:05:56.981670 6876 raft_consensus.cc:482] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:05:56.981839 6876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 1 FOLLOWER]: Advancing to term 2
I20260811 03:05:56.981900 6882 leader_election.cc:288] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:05:56.984602 6876 raft_consensus.cc:504] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39413 } }
I20260811 03:05:56.984993 6881 raft_consensus.cc:2741] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won
I20260811 03:05:56.985085 6876 leader_election.cc:302] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87d2d2f012d448faa5f212a3bd33b0a; no voters:
I20260811 03:05:56.985322 6876 leader_election.cc:288] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 03:05:56.985482 6882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 2 FOLLOWER]: Leader election won for term 2
I20260811 03:05:56.987498 6876 sys_catalog.cc:564] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:05:56.987872 6882 raft_consensus.cc:686] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [term 2 LEADER]: Becoming Leader. State: Replica: b87d2d2f012d448faa5f212a3bd33b0a, State: Running, Role: LEADER
I20260811 03:05:56.988355 6882 consensus_queue.cc:237] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39413 } }
I20260811 03:05:56.990784 6881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39413 } } }
I20260811 03:05:56.990841 6883 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b87d2d2f012d448faa5f212a3bd33b0a. Latest consensus state: current_term: 2 leader_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87d2d2f012d448faa5f212a3bd33b0a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 39413 } } }
I20260811 03:05:56.991175 6881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:56.991241 6883 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a [sys.catalog]: This master's current role is: LEADER
I20260811 03:05:56.992005 6891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:05:56.997682 6891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:05:57.011507 6891 catalog_manager.cc:1347] Generated new cluster ID: 706282c98bf14ed28b243da7cfbf65c4
I20260811 03:05:57.011669 6891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:05:57.061630 6891 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:05:57.062189 6891 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:05:57.069057 6891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b87d2d2f012d448faa5f212a3bd33b0a: Generated new TSK 0
I20260811 03:05:57.070276 6891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:05:57.088479 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39413
--builtin_ntp_servers=127.27.109.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 03:05:57.473472 6905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:57.473886 6905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:57.474053 6905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:57.482074 6905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:57.482252 6905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:57.482450 6905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:05:57.493070 6905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:57.494508 6905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:57.496513 6905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:57.511379 6910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:57.511483 6913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:57.511442 6911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:57.512602 6905 server_base.cc:1034] running on GCE node
I20260811 03:05:57.513499 6905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:57.514760 6905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:57.516038 6905 hybrid_clock.cc:648] HybridClock initialized: now 1786417557515999 us; error 32 us; skew 500 ppm
I20260811 03:05:57.516415 6905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:57.519011 6905 webserver.cc:460] Webserver started at http://127.27.109.1:45065/ using document root <none> and password file <none>
I20260811 03:05:57.519805 6905 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:57.519948 6905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:57.520306 6905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:57.523648 6905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "0512968f0c9347be8cd1549fe8fad293"
format_stamp: "Formatted at 2026-08-11 03:05:57 on dist-test-slave-8tdl"
server_key: "7ff7257901c67a08b39d94fcb1876ca8"
server_key_iv: "3009dec0375d9672411206d18824027c"
server_key_version: "encryptionkey@0"
I20260811 03:05:57.524590 6905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "0512968f0c9347be8cd1549fe8fad293"
format_stamp: "Formatted at 2026-08-11 03:05:57 on dist-test-slave-8tdl"
server_key: "7ff7257901c67a08b39d94fcb1876ca8"
server_key_iv: "3009dec0375d9672411206d18824027c"
server_key_version: "encryptionkey@0"
I20260811 03:05:57.530553 6905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260811 03:05:57.534421 6920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:57.536675 6905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:05:57.536911 6905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "0512968f0c9347be8cd1549fe8fad293"
format_stamp: "Formatted at 2026-08-11 03:05:57 on dist-test-slave-8tdl"
server_key: "7ff7257901c67a08b39d94fcb1876ca8"
server_key_iv: "3009dec0375d9672411206d18824027c"
server_key_version: "encryptionkey@0"
I20260811 03:05:57.537212 6905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:57.544149 6905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:57.545162 6905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:57.545477 6905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:57.546613 6905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:57.548295 6905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:57.548439 6905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:57.548595 6905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:57.548689 6905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:57.585496 6905 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:32791
I20260811 03:05:57.585549 7053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:32791 every 8 connection(s)
I20260811 03:05:57.587038 6905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:05:57.592620 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 6905
I20260811 03:05:57.592854 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:05:57.593534 28084 external_mini_cluster.cc:1428] Setting key 55dd0f532bec502299b7bed69bad4682
I20260811 03:05:57.596469 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39413
--builtin_ntp_servers=127.27.109.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 03:05:57.596757 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:57.589986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:38411 (local address 127.27.109.62:39413)
0811 03:05:57.590532 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:57.590536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:57.592054 (+ 1518us) server_negotiation.cc:407] Connection header received
0811 03:05:57.594084 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:57.594090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:57.594180 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:57.594314 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:57.595879 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:57.595885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:57.595890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:57.595911 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:57.595966 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:57.595979 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:57.596114 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:57.596330 (+ 216us) server_negotiation.cc:299] Negotiation successful
0811 03:05:57.596408 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":277,"threads_started":1}
I20260811 03:05:57.597756 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:57.590802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39413 (local address 127.27.109.1:38411)
0811 03:05:57.591885 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:57.591925 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:57.593782 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:57.594588 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:57.594597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:57.594765 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:57.595543 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:57.595545 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:57.595589 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:57.595694 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:57.596088 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:57.596102 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:57.596110 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 03:05:57.596423 (+ 313us) client_negotiation.cc:240] Negotiation successful
0811 03:05:57.596720 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":336,"threads_started":1}
I20260811 03:05:57.599665 7054 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39413
I20260811 03:05:57.600116 7054 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:57.600821 7054 heartbeater.cc:511] Master 127.27.109.62:39413 requested a full tablet report, sending...
I20260811 03:05:57.603600 6838 ts_manager.cc:194] Registered new tserver with Master: 0512968f0c9347be8cd1549fe8fad293 (127.27.109.1:32791)
W20260811 03:05:57.967042 7058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:57.967365 7058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:57.967459 7058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:57.975199 7058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:57.975332 7058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:57.975448 7058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:05:57.985301 7058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:57.986737 7058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:57.988893 7058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:58.003561 7065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:58.003603 7066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:58.004876 7068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:58.006040 7058 server_base.cc:1034] running on GCE node
I20260811 03:05:58.006896 7058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:58.008322 7058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:58.009737 7058 hybrid_clock.cc:648] HybridClock initialized: now 1786417558009587 us; error 71 us; skew 500 ppm
I20260811 03:05:58.010463 7058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:58.013811 7058 webserver.cc:460] Webserver started at http://127.27.109.2:37801/ using document root <none> and password file <none>
I20260811 03:05:58.014915 7058 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:58.015092 7058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:58.015494 7058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:58.019925 7058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "abd6f2a51fbd49e3ad76ba9331f09c50"
format_stamp: "Formatted at 2026-08-11 03:05:58 on dist-test-slave-8tdl"
server_key: "2e704c165904eddc90d995a31c1cf7ad"
server_key_iv: "8f3874679bb299944f5b2539812a55e5"
server_key_version: "encryptionkey@0"
I20260811 03:05:58.021268 7058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "abd6f2a51fbd49e3ad76ba9331f09c50"
format_stamp: "Formatted at 2026-08-11 03:05:58 on dist-test-slave-8tdl"
server_key: "2e704c165904eddc90d995a31c1cf7ad"
server_key_iv: "8f3874679bb299944f5b2539812a55e5"
server_key_version: "encryptionkey@0"
I20260811 03:05:58.027478 7058 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 03:05:58.031579 7075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:58.033138 7058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260811 03:05:58.033342 7058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "abd6f2a51fbd49e3ad76ba9331f09c50"
format_stamp: "Formatted at 2026-08-11 03:05:58 on dist-test-slave-8tdl"
server_key: "2e704c165904eddc90d995a31c1cf7ad"
server_key_iv: "8f3874679bb299944f5b2539812a55e5"
server_key_version: "encryptionkey@0"
I20260811 03:05:58.033674 7058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:58.040308 7058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:58.041342 7058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:58.041672 7058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:58.042918 7058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:58.044530 7058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:58.044621 7058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:58.044785 7058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:58.044883 7058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:58.082617 7058 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:35079
I20260811 03:05:58.082773 7202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:35079 every 8 connection(s)
I20260811 03:05:58.084173 7058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:05:58.091977 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 7058
I20260811 03:05:58.092168 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:05:58.092722 28084 external_mini_cluster.cc:1428] Setting key 045a663c732ec7f6baf3bf893636dd87
I20260811 03:05:58.093895 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:58.087088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:55631 (local address 127.27.109.62:39413)
0811 03:05:58.087332 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:58.087338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:58.089076 (+ 1738us) server_negotiation.cc:407] Connection header received
0811 03:05:58.090964 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:58.090969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:58.091039 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:58.091199 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:58.092930 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:58.092936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:58.092940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:58.092963 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:58.093015 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:58.093034 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:58.093192 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:58.093483 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 03:05:58.093575 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260811 03:05:58.095224 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:58.087670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39413 (local address 127.27.109.2:55631)
0811 03:05:58.088890 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:58.088933 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:58.090612 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:58.091598 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:58.091615 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:58.091742 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:58.092558 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:58.092561 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:58.092603 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:58.092698 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:58.093188 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:58.093199 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:58.093206 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 03:05:58.093593 (+ 387us) client_negotiation.cc:240] Negotiation successful
0811 03:05:58.093982 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":330,"threads_started":1}
I20260811 03:05:58.095882 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39413
--builtin_ntp_servers=127.27.109.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 03:05:58.096882 7203 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39413
I20260811 03:05:58.097260 7203 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:58.098062 7203 heartbeater.cc:511] Master 127.27.109.62:39413 requested a full tablet report, sending...
I20260811 03:05:58.099933 6838 ts_manager.cc:194] Registered new tserver with Master: abd6f2a51fbd49e3ad76ba9331f09c50 (127.27.109.2:35079)
W20260811 03:05:58.478266 7207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:05:58.478801 7207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:05:58.478943 7207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:05:58.486620 7207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:05:58.486825 7207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:05:58.486953 7207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:05:58.497038 7207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:39413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:05:58.498466 7207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:05:58.500440 7207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:05:58.515367 7216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:58.515489 7213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:05:58.515489 7214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:05:58.516731 7207 server_base.cc:1034] running on GCE node
I20260811 03:05:58.517568 7207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:05:58.518746 7207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:05:58.520040 7207 hybrid_clock.cc:648] HybridClock initialized: now 1786417558519982 us; error 42 us; skew 500 ppm
I20260811 03:05:58.520409 7207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:05:58.523216 7207 webserver.cc:460] Webserver started at http://127.27.109.3:37431/ using document root <none> and password file <none>
I20260811 03:05:58.524042 7207 fs_manager.cc:359] Metadata directory not provided
I20260811 03:05:58.524194 7207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:05:58.524574 7207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:05:58.528036 7207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "1702c05c2eec430f936a17e2d0fc0d37"
format_stamp: "Formatted at 2026-08-11 03:05:58 on dist-test-slave-8tdl"
server_key: "ef13b34fdda3ca17415235f73d91c648"
server_key_iv: "99b8b397b94f597ab46da9c630d29a6c"
server_key_version: "encryptionkey@0"
I20260811 03:05:58.528942 7207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "1702c05c2eec430f936a17e2d0fc0d37"
format_stamp: "Formatted at 2026-08-11 03:05:58 on dist-test-slave-8tdl"
server_key: "ef13b34fdda3ca17415235f73d91c648"
server_key_iv: "99b8b397b94f597ab46da9c630d29a6c"
server_key_version: "encryptionkey@0"
I20260811 03:05:58.535312 7207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260811 03:05:58.539397 7223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:58.540982 7207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:05:58.541224 7207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "1702c05c2eec430f936a17e2d0fc0d37"
format_stamp: "Formatted at 2026-08-11 03:05:58 on dist-test-slave-8tdl"
server_key: "ef13b34fdda3ca17415235f73d91c648"
server_key_iv: "99b8b397b94f597ab46da9c630d29a6c"
server_key_version: "encryptionkey@0"
I20260811 03:05:58.541566 7207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:05:58.548192 7207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:05:58.549154 7207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:05:58.549477 7207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:05:58.550519 7207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:05:58.552192 7207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:05:58.552323 7207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:58.552481 7207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:05:58.552582 7207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:05:58.593269 7207 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:41789
I20260811 03:05:58.593401 7350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:41789 every 8 connection(s)
I20260811 03:05:58.594794 7207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:05:58.601476 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 7207
I20260811 03:05:58.601675 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:05:58.602301 28084 external_mini_cluster.cc:1428] Setting key c5399965f789e03d6b781fdd17bbec62
I20260811 03:05:58.605155 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:58.597939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:60959 (local address 127.27.109.62:39413)
0811 03:05:58.598464 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:58.598472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:58.599890 (+ 1418us) server_negotiation.cc:407] Connection header received
0811 03:05:58.602004 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:58.602012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:58.602101 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:58.602248 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:58.604148 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:58.604157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:58.604163 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:58.604189 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:58.604245 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:58.604268 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:58.604388 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:58.604723 (+ 335us) server_negotiation.cc:299] Negotiation successful
0811 03:05:58.604798 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260811 03:05:58.606441 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:58.598553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:39413 (local address 127.27.109.3:60959)
0811 03:05:58.599707 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0811 03:05:58.599748 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 03:05:58.601506 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:05:58.602566 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:05:58.602576 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:05:58.602753 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:05:58.603592 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:05:58.603594 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:05:58.603641 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:05:58.603773 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:05:58.604396 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:05:58.604409 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:05:58.604418 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 03:05:58.604766 (+ 348us) client_negotiation.cc:240] Negotiation successful
0811 03:05:58.605097 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":373,"threads_started":1}
I20260811 03:05:58.607945 7054 heartbeater.cc:503] Master 127.27.109.62:39413 was elected leader, sending a full tablet report...
I20260811 03:05:58.608028 7351 heartbeater.cc:347] Connected to a master server at 127.27.109.62:39413
I20260811 03:05:58.608405 7351 heartbeater.cc:464] Registering TS with master...
I20260811 03:05:58.609248 7351 heartbeater.cc:511] Master 127.27.109.62:39413 requested a full tablet report, sending...
I20260811 03:05:58.611495 6838 ts_manager.cc:194] Registered new tserver with Master: 1702c05c2eec430f936a17e2d0fc0d37 (127.27.109.3:41789)
I20260811 03:05:58.619601 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 03:05:58.626607 7354 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:05:58.624847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.27.109.62:39413)
0811 03:05:58.625122 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:58.625132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:58.625562 (+ 430us) server_negotiation.cc:407] Connection header received
0811 03:05:58.625716 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:58.625720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:58.625774 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:58.625862 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:58.626315 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54980: BlockingRecv error: recv got EOF from 127.0.0.1:54980 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 03:05:58.627954 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 6905
I20260811 03:05:58.632140 7362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:58.629259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.27.109.1:32791)
0811 03:05:58.629898 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:58.629913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:58.629941 (+ 28us) server_negotiation.cc:407] Connection header received
0811 03:05:58.630016 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:58.630022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:58.630241 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:58.630386 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:58.631469 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:58.631476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:58.631488 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:58.631501 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:58.631544 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:58.631558 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:58.631706 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:58.631759 (+ 53us) server_negotiation.cc:299] Negotiation successful
0811 03:05:58.631849 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":327,"threads_started":1}
I20260811 03:05:58.633468 7049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:58.719064 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 6905
I20260811 03:05:58.740666 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 7058
I20260811 03:05:58.745169 7364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:58.741753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45222 (local address 127.27.109.2:35079)
0811 03:05:58.742379 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:58.742387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:58.742419 (+ 32us) server_negotiation.cc:407] Connection header received
0811 03:05:58.742494 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:58.742500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:58.742799 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:58.743174 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:58.744272 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:58.744279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:58.744291 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:58.744310 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:58.744351 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:58.744365 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:58.744533 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:58.744726 (+ 193us) server_negotiation.cc:299] Negotiation successful
0811 03:05:58.744840 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":352,"threads_started":1}
I20260811 03:05:58.746990 7198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:58.837301 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 7058
I20260811 03:05:58.861073 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 7207
I20260811 03:05:58.866500 7366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:05:58.862526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.27.109.3:41789)
0811 03:05:58.863530 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0811 03:05:58.863538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:05:58.863577 (+ 39us) server_negotiation.cc:407] Connection header received
0811 03:05:58.863656 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:05:58.863664 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:05:58.863917 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:05:58.864094 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:05:58.865681 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:05:58.865688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:05:58.865702 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:05:58.865722 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:05:58.865766 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:05:58.865781 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:05:58.865943 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 03:05:58.866093 (+ 150us) server_negotiation.cc:299] Negotiation successful
0811 03:05:58.866191 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":582,"threads_started":1}
I20260811 03:05:58.868012 7346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:58.956552 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 7207
I20260811 03:05:58.978523 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 6803
I20260811 03:05:58.979997 6871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:05:59.093397 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 6803
2026-08-11T03:05:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2713 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:05:59 dist-test-slave-8tdl krb5kdc[7372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:05:59 dist-test-slave-8tdl krb5kdc[7372](info): set up 2 sockets
Aug 11 03:05:59 dist-test-slave-8tdl krb5kdc[7372](info): commencing operation
krb5kdc: starting...
W20260811 03:06:00.322566 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.005s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:06:00 dist-test-slave-8tdl krb5kdc[7372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:06:00.659556 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38393
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:06:01.028005 7391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:06:01.028414 7391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:06:01.028594 7391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:06:01.036682 7391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:06:01.036875 7391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:06:01.036984 7391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:06:01.037092 7391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:06:01.037170 7391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:06:01.048939 7391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:06:01.050572 7391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:06:01.052101 7391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:06:01.065891 7398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:01.065785 7402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:01.065774 7399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:06:01.066915 7391 server_base.cc:1034] running on GCE node
I20260811 03:06:01.067936 7391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:06:01.069502 7391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:06:01.070822 7391 hybrid_clock.cc:648] HybridClock initialized: now 1786417561070773 us; error 31 us; skew 500 ppm
I20260811 03:06:01.071204 7391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:06:01 dist-test-slave-8tdl krb5kdc[7372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417561, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:06:01.078456 7391 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:06:01.080451 7391 webserver.cc:460] Webserver started at http://127.27.109.62:46665/ using document root <none> and password file <none>
I20260811 03:06:01.081223 7391 fs_manager.cc:359] Metadata directory not provided
I20260811 03:06:01.081344 7391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:06:01.081634 7391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:06:01.085011 7391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "411cc684f6e94b9fbca0f754e9d11747"
format_stamp: "Formatted at 2026-08-11 03:06:01 on dist-test-slave-8tdl"
server_key: "620582a7bb8461e0247607b2d9abeff5"
server_key_iv: "7229d53dd97c4b99d84f26f1785fdf91"
server_key_version: "encryptionkey@0"
I20260811 03:06:01.085827 7391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "411cc684f6e94b9fbca0f754e9d11747"
format_stamp: "Formatted at 2026-08-11 03:06:01 on dist-test-slave-8tdl"
server_key: "620582a7bb8461e0247607b2d9abeff5"
server_key_iv: "7229d53dd97c4b99d84f26f1785fdf91"
server_key_version: "encryptionkey@0"
I20260811 03:06:01.092111 7391 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260811 03:06:01.096189 7410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:01.097993 7391 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:06:01.098167 7391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "411cc684f6e94b9fbca0f754e9d11747"
format_stamp: "Formatted at 2026-08-11 03:06:01 on dist-test-slave-8tdl"
server_key: "620582a7bb8461e0247607b2d9abeff5"
server_key_iv: "7229d53dd97c4b99d84f26f1785fdf91"
server_key_version: "encryptionkey@0"
I20260811 03:06:01.098364 7391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:06:01.124202 7391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:06:01.129274 7391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:06:01.129550 7391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:06:01.150619 7391 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:38393
I20260811 03:06:01.150665 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:38393 every 8 connection(s)
I20260811 03:06:01.152240 7391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:06:01.156431 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:06:01.162755 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 7391
I20260811 03:06:01.162999 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:06:01.163900 28084 external_mini_cluster.cc:1428] Setting key 482fa88d91ae4bca0e5c2d98f381c5df
I20260811 03:06:01.168998 7465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747: Bootstrap starting.
I20260811 03:06:01.173151 7465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747: Neither blocks nor log segments found. Creating new log.
I20260811 03:06:01.174532 7465 log.cc:824] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747: Log is configured to *not* fsync() on all Append() calls
I20260811 03:06:01.178771 7465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747: No bootstrap required, opened a new log
Aug 11 03:06:01 dist-test-slave-8tdl krb5kdc[7372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:06:01.187510 7465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411cc684f6e94b9fbca0f754e9d11747" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38393 } }
I20260811 03:06:01.187984 7465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:06:01.188126 7465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411cc684f6e94b9fbca0f754e9d11747, State: Initialized, Role: FOLLOWER
I20260811 03:06:01.188956 7465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411cc684f6e94b9fbca0f754e9d11747" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38393 } }
I20260811 03:06:01.189263 7465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:06:01.189427 7465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:06:01.189631 7465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:06:01.190587 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:01.166317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.27.109.62:38393)
0811 03:06:01.167232 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:01.167245 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:01.167288 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:06:01.169130 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:01.169154 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:01.169644 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:01.170034 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:01.172091 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:01.173554 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:01.175587 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:01.176631 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:01.176782 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:01.181938 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:01.181947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:01.181964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:01.182013 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:01.185669 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:01.186561 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:01.186569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:01.186579 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:01.186848 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:01.187279 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:01.187285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:01.187288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:01.187951 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:01.188295 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:01.188804 (+ 509us) server_negotiation.cc:299] Negotiation successful
0811 03:06:01.189266 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":466,"threads_started":1}
I20260811 03:06:01.192844 7465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411cc684f6e94b9fbca0f754e9d11747" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38393 } }
I20260811 03:06:01.193403 7465 leader_election.cc:302] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 411cc684f6e94b9fbca0f754e9d11747; no voters:
I20260811 03:06:01.194095 7465 leader_election.cc:288] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:06:01.194252 7470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:06:01.195325 7470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [term 1 LEADER]: Becoming Leader. State: Replica: 411cc684f6e94b9fbca0f754e9d11747, State: Running, Role: LEADER
I20260811 03:06:01.195971 7470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411cc684f6e94b9fbca0f754e9d11747" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38393 } }
I20260811 03:06:01.196910 7465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:06:01.199084 7472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411cc684f6e94b9fbca0f754e9d11747. Latest consensus state: current_term: 1 leader_uuid: "411cc684f6e94b9fbca0f754e9d11747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411cc684f6e94b9fbca0f754e9d11747" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38393 } } }
I20260811 03:06:01.199548 7472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [sys.catalog]: This master's current role is: LEADER
I20260811 03:06:01.200915 7471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "411cc684f6e94b9fbca0f754e9d11747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411cc684f6e94b9fbca0f754e9d11747" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38393 } } }
I20260811 03:06:01.201442 7471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747 [sys.catalog]: This master's current role is: LEADER
I20260811 03:06:01.202801 7480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:06:01.208303 7480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:06:01.218425 7480 catalog_manager.cc:1347] Generated new cluster ID: f79d20d82ede453a8047516f45f5abc1
I20260811 03:06:01.218569 7480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:06:01.249095 7480 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:06:01.251219 7480 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:06:01.261412 7480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747: Generated new TSK 0
I20260811 03:06:01.262338 7480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:06:01.347693 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38393
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:06:01.733381 7496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:06:01.733748 7496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:06:01.733932 7496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:06:01.742518 7496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:06:01.742728 7496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:06:01.742919 7496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:06:01.754101 7496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:06:01.755743 7496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:06:01.757165 7496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:06:01.772154 7501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:01.772197 7504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:01.772262 7502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:06:01.773591 7496 server_base.cc:1034] running on GCE node
I20260811 03:06:01.774349 7496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:06:01.775629 7496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:06:01.776947 7496 hybrid_clock.cc:648] HybridClock initialized: now 1786417561776879 us; error 63 us; skew 500 ppm
I20260811 03:06:01.777362 7496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:06:01 dist-test-slave-8tdl krb5kdc[7372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417561, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:06:01.781256 7496 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:06:01.783150 7496 webserver.cc:460] Webserver started at http://127.27.109.1:40325/ using document root <none> and password file <none>
I20260811 03:06:01.783991 7496 fs_manager.cc:359] Metadata directory not provided
I20260811 03:06:01.784157 7496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:06:01.784512 7496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:06:01.787906 7496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "3608cb5e92ea4b99b791434ab334c1de"
format_stamp: "Formatted at 2026-08-11 03:06:01 on dist-test-slave-8tdl"
server_key: "a5945583f54747f3ebfc0213e6d920b7"
server_key_iv: "533497e081f52fe62421acf870482358"
server_key_version: "encryptionkey@0"
I20260811 03:06:01.788795 7496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "3608cb5e92ea4b99b791434ab334c1de"
format_stamp: "Formatted at 2026-08-11 03:06:01 on dist-test-slave-8tdl"
server_key: "a5945583f54747f3ebfc0213e6d920b7"
server_key_iv: "533497e081f52fe62421acf870482358"
server_key_version: "encryptionkey@0"
I20260811 03:06:01.794814 7496 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 03:06:01.799078 7515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:01.800829 7496 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:06:01.801093 7496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "3608cb5e92ea4b99b791434ab334c1de"
format_stamp: "Formatted at 2026-08-11 03:06:01 on dist-test-slave-8tdl"
server_key: "a5945583f54747f3ebfc0213e6d920b7"
server_key_iv: "533497e081f52fe62421acf870482358"
server_key_version: "encryptionkey@0"
I20260811 03:06:01.801357 7496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:06:01.823145 7496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:06:01.828400 7496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:06:01.828833 7496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:06:01.830168 7496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:06:01.831822 7496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:06:01.831954 7496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:01.832126 7496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:06:01.832203 7496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:01.874013 7496 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:34551
I20260811 03:06:01.874085 7640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:34551 every 8 connection(s)
I20260811 03:06:01.876169 7496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:06:01.883246 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 7496
I20260811 03:06:01.883497 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:06:01.884361 28084 external_mini_cluster.cc:1428] Setting key 8fbe7fa9df6d6dd9c1d62839ccf30a9d
Aug 11 03:06:01 dist-test-slave-8tdl krb5kdc[7372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417561, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:06:01.900528 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:01.879664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:52639 (local address 127.27.109.62:38393)
0811 03:06:01.880254 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:01.880259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:01.881619 (+ 1360us) server_negotiation.cc:407] Connection header received
0811 03:06:01.884057 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:01.884065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:01.884183 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:01.884326 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:01.886239 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:01.887191 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:01.889125 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:01.889588 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:01.889763 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:01.894073 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:01.894079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:01.894083 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:01.894121 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:01.896823 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:01.897842 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:01.897850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:01.897854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:01.897987 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:01.898507 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:01.898513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:01.898516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:01.898896 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:01.899059 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:01.899863 (+ 804us) server_negotiation.cc:299] Negotiation successful
0811 03:06:01.900119 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":292,"threads_started":1}
I20260811 03:06:01.901650 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:01.880359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38393 (local address 127.27.109.1:52639)
0811 03:06:01.881424 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:01.881475 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:01.883653 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:01.884653 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:01.884663 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:01.885308 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:06:01.886015 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:01.886038 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:01.887403 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:01.887408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:01.888898 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:01.888915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:01.889910 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:01.889920 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:01.890214 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:01.891079 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:06:01.891112 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:06:01.893774 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:06:01.897065 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:01.897082 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:01.897097 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:01.897710 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:01.898108 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:01.898113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:01.898117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:01.898377 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:01.899040 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:06:01.899049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:06:01.899457 (+ 408us) client_negotiation.cc:769] Sending connection context
0811 03:06:01.899747 (+ 290us) client_negotiation.cc:240] Negotiation successful
0811 03:06:01.900373 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":474,"threads_started":1}
I20260811 03:06:01.903532 7642 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38393
I20260811 03:06:01.903934 7642 heartbeater.cc:464] Registering TS with master...
I20260811 03:06:01.905010 7642 heartbeater.cc:511] Master 127.27.109.62:38393 requested a full tablet report, sending...
I20260811 03:06:01.907948 7428 ts_manager.cc:194] Registered new tserver with Master: 3608cb5e92ea4b99b791434ab334c1de (127.27.109.1:34551)
I20260811 03:06:01.909660 7428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:52639
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:06:01.960165 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38393
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:06:02.344142 7649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:06:02.344543 7649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:06:02.344728 7649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:06:02.352743 7649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:06:02.352926 7649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:06:02.353128 7649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:06:02.363916 7649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:06:02.365459 7649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:06:02.367046 7649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:06:02.381417 7655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:02.381455 7657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:02.381488 7654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:06:02.382865 7649 server_base.cc:1034] running on GCE node
I20260811 03:06:02.383605 7649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:06:02.384809 7649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:06:02.386121 7649 hybrid_clock.cc:648] HybridClock initialized: now 1786417562386068 us; error 60 us; skew 500 ppm
I20260811 03:06:02.386523 7649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:06:02 dist-test-slave-8tdl krb5kdc[7372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:06:02.390372 7649 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:06:02.392248 7649 webserver.cc:460] Webserver started at http://127.27.109.2:43267/ using document root <none> and password file <none>
I20260811 03:06:02.393054 7649 fs_manager.cc:359] Metadata directory not provided
I20260811 03:06:02.393219 7649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:06:02.393546 7649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:06:02.397002 7649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "db9d125354944e8d918d978d3ec08fe4"
format_stamp: "Formatted at 2026-08-11 03:06:02 on dist-test-slave-8tdl"
server_key: "d57e3783cc1c7980094e744767b4d704"
server_key_iv: "b21f8b7dbb41614a3ef38b93b623f6f8"
server_key_version: "encryptionkey@0"
I20260811 03:06:02.397985 7649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "db9d125354944e8d918d978d3ec08fe4"
format_stamp: "Formatted at 2026-08-11 03:06:02 on dist-test-slave-8tdl"
server_key: "d57e3783cc1c7980094e744767b4d704"
server_key_iv: "b21f8b7dbb41614a3ef38b93b623f6f8"
server_key_version: "encryptionkey@0"
I20260811 03:06:02.404809 7649 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260811 03:06:02.409181 7665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:02.410902 7649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:06:02.411129 7649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "db9d125354944e8d918d978d3ec08fe4"
format_stamp: "Formatted at 2026-08-11 03:06:02 on dist-test-slave-8tdl"
server_key: "d57e3783cc1c7980094e744767b4d704"
server_key_iv: "b21f8b7dbb41614a3ef38b93b623f6f8"
server_key_version: "encryptionkey@0"
I20260811 03:06:02.411445 7649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:06:02.431558 7649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:06:02.436563 7649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:06:02.436854 7649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:06:02.437901 7649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:06:02.439742 7649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:06:02.439824 7649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:02.439919 7649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:06:02.439963 7649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:02.477335 7649 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:36807
I20260811 03:06:02.477404 7794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:36807 every 8 connection(s)
I20260811 03:06:02.478957 7649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:06:02.484352 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 7649
I20260811 03:06:02.484613 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:06:02.485210 28084 external_mini_cluster.cc:1428] Setting key ff541da9e63653aa23645e6d4d9efd2e
Aug 11 03:06:02 dist-test-slave-8tdl krb5kdc[7372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:06:02.505297 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:02.481895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:32875 (local address 127.27.109.62:38393)
0811 03:06:02.482574 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:02.482581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:02.483648 (+ 1067us) server_negotiation.cc:407] Connection header received
0811 03:06:02.486314 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:02.486320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:02.486480 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:02.486592 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:02.488614 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:02.489806 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:02.491902 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:02.492656 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:02.492806 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:02.496832 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:02.496842 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:02.496849 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:02.496900 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:02.501008 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:02.502282 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:02.502290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:02.502293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:02.502454 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:02.503131 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:02.503135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:02.503137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:02.503529 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:02.503692 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:02.504720 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0811 03:06:02.505013 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":403,"threads_started":1}
I20260811 03:06:02.507067 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:02.482570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38393 (local address 127.27.109.2:32875)
0811 03:06:02.483458 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:02.483507 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:02.485988 (+ 2481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:02.486998 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:02.487012 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:02.487759 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:06:02.488393 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:02.488413 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:02.490024 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:02.490032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:02.491718 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:02.491734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:02.493005 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:02.493014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:02.493238 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:02.494027 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:06:02.494057 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:06:02.496527 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:06:02.501229 (+ 4702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:02.501238 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:02.501255 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:02.502076 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:02.502615 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:02.502619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:02.502624 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:02.502959 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:02.503692 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:06:02.503705 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:06:02.504229 (+ 524us) client_negotiation.cc:769] Sending connection context
0811 03:06:02.504579 (+ 350us) client_negotiation.cc:240] Negotiation successful
0811 03:06:02.505237 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":347,"threads_started":1}
I20260811 03:06:02.508893 7795 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38393
I20260811 03:06:02.509346 7795 heartbeater.cc:464] Registering TS with master...
I20260811 03:06:02.510201 7795 heartbeater.cc:511] Master 127.27.109.62:38393 requested a full tablet report, sending...
I20260811 03:06:02.512290 7428 ts_manager.cc:194] Registered new tserver with Master: db9d125354944e8d918d978d3ec08fe4 (127.27.109.2:36807)
I20260811 03:06:02.513192 7428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:32875
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:06:02.567171 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38393
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:06:02.913609 7642 heartbeater.cc:503] Master 127.27.109.62:38393 was elected leader, sending a full tablet report...
W20260811 03:06:02.959080 7801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:06:02.959466 7801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:06:02.959649 7801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:06:02.967514 7801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:06:02.967697 7801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:06:02.967823 7801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:06:02.978520 7801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:38393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:06:02.979992 7801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:06:02.981415 7801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:06:02.996255 7810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:02.996227 7807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:02.996320 7808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:06:02.997174 7801 server_base.cc:1034] running on GCE node
I20260811 03:06:02.997993 7801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:06:02.999246 7801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:06:03.000479 7801 hybrid_clock.cc:648] HybridClock initialized: now 1786417563000427 us; error 49 us; skew 500 ppm
I20260811 03:06:03.000875 7801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:06:03 dist-test-slave-8tdl krb5kdc[7372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:06:03.005659 7801 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:06:03.007725 7801 webserver.cc:460] Webserver started at http://127.27.109.3:35927/ using document root <none> and password file <none>
I20260811 03:06:03.008597 7801 fs_manager.cc:359] Metadata directory not provided
I20260811 03:06:03.008766 7801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:06:03.009145 7801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:06:03.012916 7801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "b242a5eabcbc47d989080c180dd6c9da"
format_stamp: "Formatted at 2026-08-11 03:06:03 on dist-test-slave-8tdl"
server_key: "7ce4513ff262e21ec36a9bfbf8f9bcd2"
server_key_iv: "64036ba9ffac5b5809400115c5d346ec"
server_key_version: "encryptionkey@0"
I20260811 03:06:03.013904 7801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "b242a5eabcbc47d989080c180dd6c9da"
format_stamp: "Formatted at 2026-08-11 03:06:03 on dist-test-slave-8tdl"
server_key: "7ce4513ff262e21ec36a9bfbf8f9bcd2"
server_key_iv: "64036ba9ffac5b5809400115c5d346ec"
server_key_version: "encryptionkey@0"
I20260811 03:06:03.021070 7801 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260811 03:06:03.025228 7818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:03.027007 7801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:06:03.027227 7801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "b242a5eabcbc47d989080c180dd6c9da"
format_stamp: "Formatted at 2026-08-11 03:06:03 on dist-test-slave-8tdl"
server_key: "7ce4513ff262e21ec36a9bfbf8f9bcd2"
server_key_iv: "64036ba9ffac5b5809400115c5d346ec"
server_key_version: "encryptionkey@0"
I20260811 03:06:03.027487 7801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:06:03.049296 7801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:06:03.054397 7801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:06:03.054831 7801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:06:03.055984 7801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:06:03.057574 7801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:06:03.057706 7801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:03.057910 7801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:06:03.057989 7801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:03.095152 7801 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:39869
I20260811 03:06:03.095220 7940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:39869 every 8 connection(s)
I20260811 03:06:03.096606 7801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:06:03.100860 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 7801
I20260811 03:06:03.101178 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:06:03.101953 28084 external_mini_cluster.cc:1428] Setting key 56ce7b15d848c834e940b1d1d2d396f8
Aug 11 03:06:03 dist-test-slave-8tdl krb5kdc[7372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417563, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:06:03.121104 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.099627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:40245 (local address 127.27.109.62:38393)
0811 03:06:03.100112 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.100117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.102773 (+ 2656us) server_negotiation.cc:407] Connection header received
0811 03:06:03.103776 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.103781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.103882 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.104000 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:03.106002 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.107126 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.108805 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.109272 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.109381 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.113803 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:03.113809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:03.113813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:03.113854 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:03.117167 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.118438 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.118444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.118447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.118629 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.119391 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.119396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.119398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.119685 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:03.119805 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.120596 (+ 791us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.120788 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":220,"threads_started":1}
I20260811 03:06:03.122265 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.100233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38393 (local address 127.27.109.3:40245)
0811 03:06:03.101141 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:03.101197 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:03.103482 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:03.104290 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:03.104298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:03.104899 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:06:03.105737 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.105765 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.107293 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.107297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.108630 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.108643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.109538 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.109544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.109782 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.110548 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:06:03.110581 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:06:03.113539 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:06:03.117365 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:03.117378 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:03.117399 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:03.118098 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:03.118813 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:03.118818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:03.118821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:03.119217 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:03.119807 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:06:03.119813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:06:03.120235 (+ 422us) client_negotiation.cc:769] Sending connection context
0811 03:06:03.120485 (+ 250us) client_negotiation.cc:240] Negotiation successful
0811 03:06:03.120938 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":391,"threads_started":1}
I20260811 03:06:03.123976 7942 heartbeater.cc:347] Connected to a master server at 127.27.109.62:38393
I20260811 03:06:03.124337 7942 heartbeater.cc:464] Registering TS with master...
I20260811 03:06:03.125157 7942 heartbeater.cc:511] Master 127.27.109.62:38393 requested a full tablet report, sending...
I20260811 03:06:03.126785 7428 ts_manager.cc:194] Registered new tserver with Master: b242a5eabcbc47d989080c180dd6c9da (127.27.109.3:39869)
I20260811 03:06:03.127676 7428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:40245
I20260811 03:06:03.132108 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:06:03.153024 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.137698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.27.109.62:38393)
0811 03:06:03.137898 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.137903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.138551 (+ 648us) server_negotiation.cc:407] Connection header received
0811 03:06:03.138858 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.138863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.138943 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.139051 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:03.140734 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.141675 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.143160 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.143602 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.143717 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.146266 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:03.146271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:03.146274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:03.146315 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:03.149604 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.150465 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.150471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.150474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.150660 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.151220 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.151227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.151232 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.151767 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:03.152017 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.152402 (+ 385us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.152676 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:06:03.158079 7428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:06:03.166668 7428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:06:03.206836 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.182605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43006 (local address 127.27.109.1:34551)
0811 03:06:03.193794 (+ 11189us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.193808 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.197776 (+ 3968us) server_negotiation.cc:407] Connection header received
0811 03:06:03.197896 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.197905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.198144 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.198368 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:03.199014 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.200162 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.203699 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.205178 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.205341 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.205607 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.206242 (+ 635us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.206437 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10952,"thread_start_us":10772,"threads_started":1}
I20260811 03:06:03.206990 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.182091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34551 (local address 127.0.0.1:43006)
0811 03:06:03.197478 (+ 15387us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:03.197500 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:03.197647 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:03.198359 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:03.198364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:03.198392 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:03.198836 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.198846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.200399 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.200404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.203451 (+ 3047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.203472 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.205597 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.205608 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.205924 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.206000 (+ 76us) client_negotiation.cc:769] Sending connection context
0811 03:06:03.206155 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 03:06:03.206397 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15235,"mutex_wait_us":394,"thread_start_us":13863,"threads_started":1}
I20260811 03:06:03.207511 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.181381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.27.109.3:39869)
0811 03:06:03.196541 (+ 15160us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.196552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.196594 (+ 42us) server_negotiation.cc:407] Connection header received
0811 03:06:03.197111 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.197123 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.197422 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.197610 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:03.198766 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.200568 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.204150 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.205688 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.205818 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.206068 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.206755 (+ 687us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.207091 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14946,"thread_start_us":15331,"threads_started":1}
I20260811 03:06:03.207860 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.181138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39869 (local address 127.0.0.1:36348)
0811 03:06:03.196248 (+ 15110us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:03.196301 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:03.196527 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:03.197858 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:03.197862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:03.197887 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:03.198471 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.198488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.200812 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.200823 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.203961 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.203980 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.206018 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.206027 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.206217 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.206257 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 03:06:03.206344 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 03:06:03.206471 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14716,"mutex_wait_us":88,"spinlock_wait_cycles":2432,"thread_start_us":15558,"threads_started":1}
I20260811 03:06:03.210562 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.182956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.27.109.2:36807)
0811 03:06:03.194317 (+ 11361us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.194329 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.196731 (+ 2402us) server_negotiation.cc:407] Connection header received
0811 03:06:03.196875 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.196888 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.197220 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.197440 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:03.198331 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.200278 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.206212 (+ 5934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.209294 (+ 3082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.209461 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.209675 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.210055 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.210238 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11191,"thread_start_us":11081,"threads_started":1}
I20260811 03:06:03.210584 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.181770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:36807 (local address 127.0.0.1:45670)
0811 03:06:03.196300 (+ 14530us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:03.196323 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:03.196524 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:03.197473 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:03.197481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:03.197512 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:03.198059 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.198074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.200484 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.200493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.203962 (+ 3469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.203980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.209585 (+ 5605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.209596 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.209809 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.209849 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 03:06:03.209978 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 03:06:03.210147 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14375,"mutex_wait_us":156,"spinlock_wait_cycles":7808,"thread_start_us":15376,"threads_started":1}
I20260811 03:06:03.211918 7569 tablet_service.cc:1467] Processing CreateTablet for tablet 29e4d979316f457490ce8b8bf1ba3767 (DEFAULT_TABLE table=test-table [id=4ba9f1fafed54832b969fd7ede5171d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:06:03.211845 7868 tablet_service.cc:1467] Processing CreateTablet for tablet 29e4d979316f457490ce8b8bf1ba3767 (DEFAULT_TABLE table=test-table [id=4ba9f1fafed54832b969fd7ede5171d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:06:03.213295 7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e4d979316f457490ce8b8bf1ba3767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:06:03.213429 7868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e4d979316f457490ce8b8bf1ba3767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:06:03.214948 7717 tablet_service.cc:1467] Processing CreateTablet for tablet 29e4d979316f457490ce8b8bf1ba3767 (DEFAULT_TABLE table=test-table [id=4ba9f1fafed54832b969fd7ede5171d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:06:03.216662 7717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e4d979316f457490ce8b8bf1ba3767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:06:03.226986 7963 tablet_bootstrap.cc:492] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Bootstrap starting.
I20260811 03:06:03.231637 7964 tablet_bootstrap.cc:492] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Bootstrap starting.
I20260811 03:06:03.231681 7963 tablet_bootstrap.cc:654] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Neither blocks nor log segments found. Creating new log.
I20260811 03:06:03.233261 7963 log.cc:824] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Log is configured to *not* fsync() on all Append() calls
I20260811 03:06:03.233685 7965 tablet_bootstrap.cc:492] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Bootstrap starting.
I20260811 03:06:03.236313 7964 tablet_bootstrap.cc:654] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Neither blocks nor log segments found. Creating new log.
I20260811 03:06:03.237232 7965 tablet_bootstrap.cc:654] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Neither blocks nor log segments found. Creating new log.
I20260811 03:06:03.237828 7963 tablet_bootstrap.cc:492] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: No bootstrap required, opened a new log
I20260811 03:06:03.237892 7964 log.cc:824] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Log is configured to *not* fsync() on all Append() calls
I20260811 03:06:03.238178 7963 ts_tablet_manager.cc:1397] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260811 03:06:03.238546 7965 log.cc:824] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Log is configured to *not* fsync() on all Append() calls
I20260811 03:06:03.242312 7964 tablet_bootstrap.cc:492] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: No bootstrap required, opened a new log
I20260811 03:06:03.242717 7964 ts_tablet_manager.cc:1397] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260811 03:06:03.244663 7965 tablet_bootstrap.cc:492] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: No bootstrap required, opened a new log
I20260811 03:06:03.244964 7965 ts_tablet_manager.cc:1397] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260811 03:06:03.247882 7963 raft_consensus.cc:348] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.248371 7963 raft_consensus.cc:374] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:06:03.248492 7963 raft_consensus.cc:729] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3608cb5e92ea4b99b791434ab334c1de, State: Initialized, Role: FOLLOWER
I20260811 03:06:03.249230 7963 consensus_queue.cc:260] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.251354 7963 ts_tablet_manager.cc:1428] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260811 03:06:03.253943 7964 raft_consensus.cc:348] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.254498 7964 raft_consensus.cc:374] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:06:03.254639 7964 raft_consensus.cc:729] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b242a5eabcbc47d989080c180dd6c9da, State: Initialized, Role: FOLLOWER
I20260811 03:06:03.255425 7964 consensus_queue.cc:260] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.256789 7942 heartbeater.cc:503] Master 127.27.109.62:38393 was elected leader, sending a full tablet report...
I20260811 03:06:03.257666 7964 ts_tablet_manager.cc:1428] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260811 03:06:03.257995 7965 raft_consensus.cc:348] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.258597 7965 raft_consensus.cc:374] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:06:03.258822 7965 raft_consensus.cc:729] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db9d125354944e8d918d978d3ec08fe4, State: Initialized, Role: FOLLOWER
I20260811 03:06:03.259529 7965 consensus_queue.cc:260] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.260820 7795 heartbeater.cc:503] Master 127.27.109.62:38393 was elected leader, sending a full tablet report...
I20260811 03:06:03.261701 7965 ts_tablet_manager.cc:1428] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260811 03:06:03.348557 7943 tablet.cc:2369] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:06:03.379451 7643 tablet.cc:2369] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:06:03.417447 7971 raft_consensus.cc:482] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:06:03.417822 7971 raft_consensus.cc:504] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.419914 7971 leader_election.cc:288] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b242a5eabcbc47d989080c180dd6c9da (127.27.109.3:39869), 3608cb5e92ea4b99b791434ab334c1de (127.27.109.1:34551)
I20260811 03:06:03.430327 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.420550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:58089 (local address 127.27.109.1:34551)
0811 03:06:03.420763 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.420770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.421717 (+ 947us) server_negotiation.cc:407] Connection header received
0811 03:06:03.421889 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.421894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.421983 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.422099 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:03.423186 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.424390 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.427832 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.429087 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.429257 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.429355 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.429909 (+ 554us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.430049 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 03:06:03.430325 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.420543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:52075 (local address 127.27.109.3:39869)
0811 03:06:03.420763 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.420770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.421683 (+ 913us) server_negotiation.cc:407] Connection header received
0811 03:06:03.421953 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.421960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.422048 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.422143 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:03.423241 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.424474 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.427869 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.429148 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.429363 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.429463 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.429864 (+ 401us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.430029 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 03:06:03.430853 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.420416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39869 (local address 127.27.109.2:52075)
0811 03:06:03.421510 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:03.421531 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:03.421727 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:03.422224 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:03.422229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:03.422285 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:03.423066 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.423081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.424618 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.424624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.427601 (+ 2977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.427612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.429434 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.429447 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.429632 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.429680 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 03:06:03.429799 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 03:06:03.429978 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":243,"spinlock_wait_cycles":52224,"thread_start_us":611,"threads_started":1}
I20260811 03:06:03.431092 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.420427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34551 (local address 127.27.109.2:58089)
0811 03:06:03.421597 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:03.421619 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:03.421737 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:03.422224 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:03.422229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:03.422286 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:03.422917 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.422934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.424620 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.424624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.427591 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:03.427612 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.429360 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.429368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:03.429636 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.429677 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 03:06:03.429860 (+ 183us) client_negotiation.cc:240] Negotiation successful
0811 03:06:03.430029 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1046,"mutex_wait_us":322,"spinlock_wait_cycles":47360,"thread_start_us":597,"threads_started":1}
I20260811 03:06:03.432207 7890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e4d979316f457490ce8b8bf1ba3767" candidate_uuid: "db9d125354944e8d918d978d3ec08fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b242a5eabcbc47d989080c180dd6c9da" is_pre_election: true
I20260811 03:06:03.432219 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e4d979316f457490ce8b8bf1ba3767" candidate_uuid: "db9d125354944e8d918d978d3ec08fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3608cb5e92ea4b99b791434ab334c1de" is_pre_election: true
I20260811 03:06:03.432873 7890 raft_consensus.cc:2454] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db9d125354944e8d918d978d3ec08fe4 in term 0.
I20260811 03:06:03.432871 7590 raft_consensus.cc:2454] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db9d125354944e8d918d978d3ec08fe4 in term 0.
I20260811 03:06:03.433665 7668 leader_election.cc:302] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3608cb5e92ea4b99b791434ab334c1de, b242a5eabcbc47d989080c180dd6c9da, db9d125354944e8d918d978d3ec08fe4; no voters:
I20260811 03:06:03.434158 7971 raft_consensus.cc:2781] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:06:03.434325 7971 raft_consensus.cc:482] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:06:03.434473 7971 raft_consensus.cc:3037] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:06:03.436901 7971 raft_consensus.cc:504] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.437667 7971 leader_election.cc:288] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [CANDIDATE]: Term 1 election: Requested vote from peers b242a5eabcbc47d989080c180dd6c9da (127.27.109.3:39869), 3608cb5e92ea4b99b791434ab334c1de (127.27.109.1:34551)
I20260811 03:06:03.438251 7890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e4d979316f457490ce8b8bf1ba3767" candidate_uuid: "db9d125354944e8d918d978d3ec08fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b242a5eabcbc47d989080c180dd6c9da"
I20260811 03:06:03.438344 7590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e4d979316f457490ce8b8bf1ba3767" candidate_uuid: "db9d125354944e8d918d978d3ec08fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3608cb5e92ea4b99b791434ab334c1de"
I20260811 03:06:03.438484 7890 raft_consensus.cc:3037] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:06:03.438635 7590 raft_consensus.cc:3037] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:06:03.440982 7890 raft_consensus.cc:2454] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db9d125354944e8d918d978d3ec08fe4 in term 1.
I20260811 03:06:03.441179 7590 raft_consensus.cc:2454] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db9d125354944e8d918d978d3ec08fe4 in term 1.
I20260811 03:06:03.441527 7668 leader_election.cc:302] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b242a5eabcbc47d989080c180dd6c9da, db9d125354944e8d918d978d3ec08fe4; no voters:
I20260811 03:06:03.441982 7971 raft_consensus.cc:2781] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:06:03.442461 7971 raft_consensus.cc:686] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 1 LEADER]: Becoming Leader. State: Replica: db9d125354944e8d918d978d3ec08fe4, State: Running, Role: LEADER
I20260811 03:06:03.443048 7971 consensus_queue.cc:237] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } }
I20260811 03:06:03.448314 7428 catalog_manager.cc:5495] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to db9d125354944e8d918d978d3ec08fe4 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "db9d125354944e8d918d978d3ec08fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db9d125354944e8d918d978d3ec08fe4" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36807 } health_report { overall_health: HEALTHY } } }
W20260811 03:06:03.481374 7796 tablet.cc:2369] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:06:03.500797 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.494661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.27.109.2:36807)
0811 03:06:03.494979 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.494984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.495003 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:06:03.495093 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.495097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.495183 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.495324 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:06:03.495727 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.496529 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.498043 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.498523 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.498641 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.499342 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:06:03.500008 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:06:03.500139 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.500325 (+ 186us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.500474 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 03:06:03.511090 7590 raft_consensus.cc:1261] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 1 FOLLOWER]: Refusing update from remote peer db9d125354944e8d918d978d3ec08fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:06:03.511097 7890 raft_consensus.cc:1261] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 1 FOLLOWER]: Refusing update from remote peer db9d125354944e8d918d978d3ec08fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:06:03.512497 7971 consensus_queue.cc:1035] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3608cb5e92ea4b99b791434ab334c1de" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:06:03.512902 7974 consensus_queue.cc:1035] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b242a5eabcbc47d989080c180dd6c9da" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:06:03.548067 7428 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:06:03.548421 7428 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:06:03.553208 7428 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 411cc684f6e94b9fbca0f754e9d11747: Sending DeleteTablet for 3 replicas of tablet 29e4d979316f457490ce8b8bf1ba3767
I20260811 03:06:03.554874 7569 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e4d979316f457490ce8b8bf1ba3767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:06:03 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:43006
I20260811 03:06:03.554879 7868 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e4d979316f457490ce8b8bf1ba3767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:06:03 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36348
I20260811 03:06:03.554888 7717 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e4d979316f457490ce8b8bf1ba3767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:06:03 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45670
I20260811 03:06:03.560673 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 7496
I20260811 03:06:03.567546 7987 tablet_replica.cc:330] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: stopping tablet replica
I20260811 03:06:03.567710 7989 tablet_replica.cc:330] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: stopping tablet replica
I20260811 03:06:03.568251 7987 raft_consensus.cc:2229] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:06:03.568259 7989 raft_consensus.cc:2229] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:06:03.569046 7987 raft_consensus.cc:2258] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:06:03.569072 7989 raft_consensus.cc:2258] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:06:03.573086 7987 ts_tablet_manager.cc:1905] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:06:03.575309 7988 tablet_replica.cc:330] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: stopping tablet replica
I20260811 03:06:03.575875 7988 raft_consensus.cc:2229] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:06:03.577163 7989 ts_tablet_manager.cc:1905] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:06:03.577868 7988 raft_consensus.cc:2258] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:06:03.580338 7988 ts_tablet_manager.cc:1905] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:06:03.585112 7989 ts_tablet_manager.cc:1918] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:06:03.585397 7989 log.cc:1196] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/29e4d979316f457490ce8b8bf1ba3767
I20260811 03:06:03.586198 7989 ts_tablet_manager.cc:1939] T 29e4d979316f457490ce8b8bf1ba3767 P db9d125354944e8d918d978d3ec08fe4: Deleting consensus metadata
I20260811 03:06:03.588404 7414 catalog_manager.cc:4841] TS db9d125354944e8d918d978d3ec08fe4 (127.27.109.2:36807): tablet 29e4d979316f457490ce8b8bf1ba3767 (table test-table [id=4ba9f1fafed54832b969fd7ede5171d7]) successfully deleted
I20260811 03:06:03.588980 7988 ts_tablet_manager.cc:1918] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:06:03.589270 7988 log.cc:1196] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/29e4d979316f457490ce8b8bf1ba3767
I20260811 03:06:03.589913 7988 ts_tablet_manager.cc:1939] T 29e4d979316f457490ce8b8bf1ba3767 P 3608cb5e92ea4b99b791434ab334c1de: Deleting consensus metadata
I20260811 03:06:03.592170 7411 catalog_manager.cc:4841] TS 3608cb5e92ea4b99b791434ab334c1de (127.27.109.1:34551): tablet 29e4d979316f457490ce8b8bf1ba3767 (table test-table [id=4ba9f1fafed54832b969fd7ede5171d7]) successfully deleted
Aug 11 03:06:03 dist-test-slave-8tdl krb5kdc[7372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:06:03.593999 7987 ts_tablet_manager.cc:1918] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:06:03.594211 7987 log.cc:1196] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/29e4d979316f457490ce8b8bf1ba3767
I20260811 03:06:03.594856 7987 ts_tablet_manager.cc:1939] T 29e4d979316f457490ce8b8bf1ba3767 P b242a5eabcbc47d989080c180dd6c9da: Deleting consensus metadata
I20260811 03:06:03.597193 7413 catalog_manager.cc:4841] TS b242a5eabcbc47d989080c180dd6c9da (127.27.109.3:39869): tablet 29e4d979316f457490ce8b8bf1ba3767 (table test-table [id=4ba9f1fafed54832b969fd7ede5171d7]) successfully deleted
I20260811 03:06:03.601974 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.563163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.27.109.1:34551)
0811 03:06:03.563441 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.563452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.579295 (+ 15843us) server_negotiation.cc:407] Connection header received
0811 03:06:03.579540 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.579545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.579642 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.579757 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:03.581843 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.583410 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.585810 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.586727 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.586914 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.593825 (+ 6911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:03.593844 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:03.593864 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:03.593932 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:03.598227 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.599346 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.599355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.599358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.599595 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.600187 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.600191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.600194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.600587 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:03.600831 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.601298 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.601619 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260811 03:06:03.602866 7636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:06:03.742919 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 7496
I20260811 03:06:03.765836 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 7649
Aug 11 03:06:03 dist-test-slave-8tdl krb5kdc[7372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:06:03.787917 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.766773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.27.109.2:36807)
0811 03:06:03.766967 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.766975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.766997 (+ 22us) server_negotiation.cc:407] Connection header received
0811 03:06:03.767248 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.767258 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.767329 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.767458 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:03.768908 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.769734 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.771504 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.772303 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.772440 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.779304 (+ 6864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:03.779335 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:03.779365 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:03.779485 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:03.784062 (+ 4577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.785185 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.785193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.785196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.785369 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.786001 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.786007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.786009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.786508 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:03.786953 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.787313 (+ 360us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.787520 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 03:06:03.788844 7790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:06:03.919412 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 7649
I20260811 03:06:03.940948 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 7801
Aug 11 03:06:03 dist-test-slave-8tdl krb5kdc[7372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:06:03.959194 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:03.942082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.27.109.3:39869)
0811 03:06:03.942583 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:03.942588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:03.942607 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:06:03.942666 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:03.942669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:03.942819 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:03.942916 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:03.944514 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.945260 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.946576 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:03.947233 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:03.947339 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:03.952480 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:03.952499 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:03.952517 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:03.952563 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:03.955958 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.956816 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.956826 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.956828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.956996 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:03.957450 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:03.957454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:03.957456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:03.957836 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:03.958158 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:03.958489 (+ 331us) server_negotiation.cc:299] Negotiation successful
0811 03:06:03.958771 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":265,"threads_started":1}
I20260811 03:06:03.959939 7936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:06:04.061542 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 7801
I20260811 03:06:04.083801 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 7391
I20260811 03:06:04.085166 7460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:06:04.201207 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 7391
2026-08-11T03:06:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5106 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:06:04 dist-test-slave-8tdl krb5kdc[7999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:06:04 dist-test-slave-8tdl krb5kdc[7999](info): set up 2 sockets
Aug 11 03:06:04 dist-test-slave-8tdl krb5kdc[7999](info): commencing operation
krb5kdc: starting...
W20260811 03:06:05.429839 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.005s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:06:05 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:06:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/rangeradmin_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/rangeradmin_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/rangerlookup_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/rangerlookup_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
I20260811 03:06:05.928726 28084 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWXDOoZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/postgres -l logfile start
2026-08-11 03:06:08.891 UTC [8031] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 03:06:08.891 UTC [8031] LOG: listening on IPv4 address "127.27.109.20", port 37739
2026-08-11 03:06:08.895 UTC [8031] LOG: listening on Unix socket "/tmp/.s.PGSQL.37739"
2026-08-11 03:06:08.922 UTC [8033] LOG: database system was shut down at 2026-08-11 03:06:07 UTC
2026-08-11 03:06:08.927 UTC [8031] LOG: database system is ready to accept connections
I20260811 03:06:10.031400 28084 mini_postgres.cc:96] Postgres bound to 37739
2026-08-11 03:06:10.059 UTC [8044] FATAL: database "slave" does not exist
127.27.109.20:37739 - accepting connections
I20260811 03:06:10.060827 28084 mini_ranger.cc:161] Starting Ranger...
I20260811 03:06:10.100517 28084 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 03:06:10.753082 28084 mini_ranger.cc:87] Created ranger Postgres database
I20260811 03:06:10.753378 28084 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-admin
2026-08-11 03:06:11,186 [I] DB FLAVOR :POSTGRES
2026-08-11 03:06:11,187 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 03:06:11,187 [I] Checking connection..
2026-08-11 03:06:11,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 03:06:11,495 [I] Checking connection passed.
2026-08-11 03:06:11,495 [I] --------- Verifying version history table ---------
2026-08-11 03:06:11,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 03:06:11,788 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 03:06:11,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 03:06:12,084 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 03:06:12,085 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 03:06:12,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 03:06:12.341 UTC [8115] WARNING: there is no transaction in progress
2026-08-11 03:06:12,350 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 03:06:12,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 03:06:12,652 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 03:06:12,653 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 03:06:12,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 03:06:12,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 03:06:13,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8tdl.c.gcp-upstream.internal','N') ;"
2026-08-11 03:06:13,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 03:06:13,756 [I] Table x_portal_user does not exist in database ranger
2026-08-11 03:06:13,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 03:06:14,041 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 03:06:14,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 03:06:14,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 03:06:14,587 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 03:06:14,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 03:06:15.738 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.760 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.780 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.802 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.823 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.845 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.914 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.924 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.943 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.956 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.969 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.986 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:15.998 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:16.007 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:16.019 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:16.028 UTC [8238] WARNING: there is no transaction in progress
2026-08-11 03:06:16,887 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 03:06:16,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 03:06:17,170 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 03:06:17,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 03:06:17,427 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 03:06:17,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 03:06:17,714 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 03:06:17,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 03:06:17,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 03:06:18,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8tdl.c.gcp-upstream.internal';"
2026-08-11 03:06:18,516 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 03:06:18,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 03:06:18,812 [I] DB_PATCHES have already been applied
I20260811 03:06:18.819759 28084 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWXDOoZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 03:06:18.819993 28084 mini_ranger.cc:193] Using host: 127.27.109.20
I20260811 03:06:18.846102 28084 mini_ranger.cc:227] Ranger admin URL: http://127.27.109.20:54675
Aug 11, 2026 3:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 3:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 3:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 3:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 3:06:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 3:06:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 3:06:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 03:06:20.120810 28084 mini_ranger.cc:160] Time spent starting Ranger: real 10.060s user 0.003s sys 0.077s
Aug 11, 2026 3:06:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 3:06:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 3:06:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 3:06:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 3:06:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 03:06:43.829926 28084 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/rangerkms_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/rangerkms_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.109.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 03:06:44 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417604, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 03:06:44.076647 28084 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 03:06:44.107862 28084 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 03:06:45.172951 28084 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 03:06:45.173113 28084 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms
I20260811 03:06:45.173271 28084 mini_ranger_kms.cc:227] Using postgres at 127.27.109.20:37739
2026-08-11 03:06:46,157 [I] DB FLAVOR :POSTGRES
2026-08-11 03:06:46,157 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 03:06:46,157 [I] Checking connection
2026-08-11 03:06:46,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 03:06:46,419 [I] connection success
2026-08-11 03:06:46,419 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 03:06:46,419 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 03:06:46,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 03:06:46,663 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 03:06:46,664 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 03:06:46,664 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 03:06:46,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:37739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 03:06:46,955 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 03:06:47.465930 28084 mini_ranger_kms.cc:310] Created kms service
I20260811 03:06:47.695693 28084 mini_ranger_kms.cc:326] Created kudu user
I20260811 03:06:47.745270 28084 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 03:06:47 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417604, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:06:48.200889 28084 mini_ranger_kms.cc:384] Added ranger policy
I20260811 03:06:48.202332 28084 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 03:06:48.202441 28084 mini_ranger_kms.cc:240] Using host: 127.27.109.20
I20260811 03:06:48.235212 28084 mini_ranger_kms.cc:276] Ranger KMS PID: 8443
I20260811 03:06:48.235440 28084 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.109.20:53375
Aug 11, 2026 3:06:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 3:06:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 3:06:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 3:06:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 3:06:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 3:06:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260811 03:06:50.883605 28084 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.807s user 0.163s sys 0.516s
I20260811 03:06:50.883999 28084 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 03:06:50.884460 28084 mini_ranger_kms.cc:401] 127.27.109.20:53375/kms/v1/keys
Aug 11 03:06:51 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786417611, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 11 03:06:51 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417611, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 11 03:06:52 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417611, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 11 03:06:54 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:06:54.666409 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:43839
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:43839
--ranger_config_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:53375/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:06:55.052556 8498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:06:55.052955 8498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:06:55.053136 8498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:06:55.060765 8498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:06:55.060953 8498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:06:55.061034 8498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:06:55.061121 8498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:06:55.061216 8498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:06:55.072098 8498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.109.20:53375/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:43839
--ranger_config_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:43839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:06:55.073666 8498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:06:55.075352 8498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:06:55.088747 8504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:55.088836 8505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:55.088871 8507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:06:55.090090 8498 server_base.cc:1034] running on GCE node
I20260811 03:06:55.091042 8498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:06:55.092636 8498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:06:55.093891 8498 hybrid_clock.cc:648] HybridClock initialized: now 1786417615093839 us; error 43 us; skew 500 ppm
I20260811 03:06:55.094370 8498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:06:55 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:06:55.097863 8498 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:06:55.099713 8498 webserver.cc:460] Webserver started at http://127.27.109.62:35147/ using document root <none> and password file <none>
I20260811 03:06:55.100468 8498 fs_manager.cc:359] Metadata directory not provided
I20260811 03:06:55.100623 8498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:06:55.100968 8498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 03:06:55 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417615, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:06:55.158126 8498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "b27ae54eac404602ab578dbe8c5712cf"
format_stamp: "Formatted at 2026-08-11 03:06:55 on dist-test-slave-8tdl"
server_key: "11db576dfc5e047db7191904a17ef330"
server_key_iv: "446be47c50887e25c15e12ff427357d0"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:55.159384 8498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "b27ae54eac404602ab578dbe8c5712cf"
format_stamp: "Formatted at 2026-08-11 03:06:55 on dist-test-slave-8tdl"
server_key: "11db576dfc5e047db7191904a17ef330"
server_key_iv: "446be47c50887e25c15e12ff427357d0"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:55.167131 8498 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260811 03:06:55.190524 8515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:55.192567 8498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 03:06:55.192785 8498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "b27ae54eac404602ab578dbe8c5712cf"
format_stamp: "Formatted at 2026-08-11 03:06:55 on dist-test-slave-8tdl"
server_key: "11db576dfc5e047db7191904a17ef330"
server_key_iv: "446be47c50887e25c15e12ff427357d0"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:55.193059 8498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:06:55.225499 8498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:06:55.230187 8498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:06:55.230597 8498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:06:55.253613 8498 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:43839
I20260811 03:06:55.253661 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:43839 every 8 connection(s)
I20260811 03:06:55.255170 8498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:06:55.256392 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 8498
I20260811 03:06:55.256626 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:06:55.257490 28084 external_mini_cluster.cc:1428] Setting key 3bf17d47d6742e579d33332e8b54d91a
I20260811 03:06:55.260470 8571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 03:06:55 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:06:55.275255 8571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf: Bootstrap starting.
I20260811 03:06:55.279904 8571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf: Neither blocks nor log segments found. Creating new log.
I20260811 03:06:55.281433 8571 log.cc:824] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf: Log is configured to *not* fsync() on all Append() calls
I20260811 03:06:55.285305 8571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf: No bootstrap required, opened a new log
I20260811 03:06:55.285650 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:55.259946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.27.109.62:43839)
0811 03:06:55.260853 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:55.260869 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:55.260922 (+ 53us) server_negotiation.cc:407] Connection header received
0811 03:06:55.263620 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:55.263651 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:55.264250 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:55.264743 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:55.266147 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:55.268339 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:55.269782 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:55.271009 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:55.271220 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:55.275698 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:55.275712 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:55.275732 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:55.275793 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:55.280191 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:55.281039 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:55.281049 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:55.281058 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:55.281297 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:55.281954 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:55.281958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:55.281961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:55.282896 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:55.283243 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:55.283985 (+ 742us) server_negotiation.cc:299] Negotiation successful
0811 03:06:55.284384 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":412,"threads_started":1}
I20260811 03:06:55.293694 8571 raft_consensus.cc:348] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27ae54eac404602ab578dbe8c5712cf" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43839 } }
I20260811 03:06:55.294133 8571 raft_consensus.cc:374] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:06:55.294251 8571 raft_consensus.cc:729] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27ae54eac404602ab578dbe8c5712cf, State: Initialized, Role: FOLLOWER
I20260811 03:06:55.294903 8571 consensus_queue.cc:260] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27ae54eac404602ab578dbe8c5712cf" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43839 } }
I20260811 03:06:55.295173 8571 raft_consensus.cc:388] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:06:55.295318 8571 raft_consensus.cc:482] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:06:55.295495 8571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:06:55.298125 8571 raft_consensus.cc:504] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27ae54eac404602ab578dbe8c5712cf" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43839 } }
I20260811 03:06:55.298651 8571 leader_election.cc:302] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b27ae54eac404602ab578dbe8c5712cf; no voters:
I20260811 03:06:55.299489 8571 leader_election.cc:288] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:06:55.299719 8579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:06:55.301389 8579 raft_consensus.cc:686] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [term 1 LEADER]: Becoming Leader. State: Replica: b27ae54eac404602ab578dbe8c5712cf, State: Running, Role: LEADER
I20260811 03:06:55.302000 8579 consensus_queue.cc:237] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27ae54eac404602ab578dbe8c5712cf" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43839 } }
I20260811 03:06:55.302537 8571 sys_catalog.cc:564] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:06:55.305212 8580 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b27ae54eac404602ab578dbe8c5712cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27ae54eac404602ab578dbe8c5712cf" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43839 } } }
I20260811 03:06:55.305683 8580 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [sys.catalog]: This master's current role is: LEADER
I20260811 03:06:55.305236 8581 sys_catalog.cc:455] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b27ae54eac404602ab578dbe8c5712cf. Latest consensus state: current_term: 1 leader_uuid: "b27ae54eac404602ab578dbe8c5712cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b27ae54eac404602ab578dbe8c5712cf" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 43839 } } }
I20260811 03:06:55.306921 8581 sys_catalog.cc:458] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf [sys.catalog]: This master's current role is: LEADER
I20260811 03:06:55.308493 8588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:06:55.313921 8588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:06:55.320459 8571 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 03:06:55.324854 8588 catalog_manager.cc:1347] Generated new cluster ID: b2f23d7cc1e74bcf855c98afae5b4533
I20260811 03:06:55.325062 8588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:06:55.367996 8588 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:06:55.370545 8588 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:06:55.383584 8588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf: Generated new TSK 0
I20260811 03:06:55.384925 8588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:06:56.259771 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:53375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43839
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:06:56 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786417616, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 03:06:56.726899 8620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:06:56.727483 8620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:06:56.727697 8620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:06:56.740155 8620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:06:56.740463 8620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:06:56.740687 8620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:06:56.757985 8620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.109.20:53375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:06:56.759900 8620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:06:56.761772 8620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:06:56.779670 8631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:56.780488 8632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:56.781091 8634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:06:56.789673 8620 server_base.cc:1034] running on GCE node
I20260811 03:06:56.790740 8620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:06:56.801925 8620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:06:56.806209 8620 hybrid_clock.cc:648] HybridClock initialized: now 1786417616806024 us; error 73 us; skew 500 ppm
I20260811 03:06:56.807073 8620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:06:56 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417616, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:06:56.812073 8620 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:06:56.818532 8620 webserver.cc:460] Webserver started at http://127.27.109.1:36321/ using document root <none> and password file <none>
I20260811 03:06:56.819473 8620 fs_manager.cc:359] Metadata directory not provided
I20260811 03:06:56.819648 8620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:06:56.820201 8620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 03:06:56 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417616, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:06:56.846470 8620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "529e475299d64c54926679f795219754"
format_stamp: "Formatted at 2026-08-11 03:06:56 on dist-test-slave-8tdl"
server_key: "d03a05f92490f3891423989cb1bb526f"
server_key_iv: "4a760c1cdbcfc8959bd7817dc3e49b4b"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:56.847836 8620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "529e475299d64c54926679f795219754"
format_stamp: "Formatted at 2026-08-11 03:06:56 on dist-test-slave-8tdl"
server_key: "d03a05f92490f3891423989cb1bb526f"
server_key_iv: "4a760c1cdbcfc8959bd7817dc3e49b4b"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:56.855463 8620 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260811 03:06:56.884822 8642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:56.887161 8620 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.008s sys 0.000s
I20260811 03:06:56.887396 8620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "529e475299d64c54926679f795219754"
format_stamp: "Formatted at 2026-08-11 03:06:56 on dist-test-slave-8tdl"
server_key: "d03a05f92490f3891423989cb1bb526f"
server_key_iv: "4a760c1cdbcfc8959bd7817dc3e49b4b"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:56.887671 8620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:06:56.924794 8620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:06:56.930995 8620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:06:56.931380 8620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:06:56.932901 8620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:06:56.936811 8620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:06:56.936993 8620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 03:06:56.937150 8620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:06:56.937217 8620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:57.019095 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:34049 every 8 connection(s)
I20260811 03:06:57.019095 8620 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:34049
I20260811 03:06:57.021335 8620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:06:57.026646 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 8620
I20260811 03:06:57.026952 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:06:57.028060 28084 external_mini_cluster.cc:1428] Setting key fa102fd30ebad9a33e09b2b69b917845
Aug 11 03:06:57 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417616, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:06:57.090623 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:57.026607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:51775 (local address 127.27.109.62:43839)
0811 03:06:57.063883 (+ 37276us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:57.063893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:57.063924 (+ 31us) server_negotiation.cc:407] Connection header received
0811 03:06:57.064016 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:57.064022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:57.064143 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:57.064285 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:57.069516 (+ 5231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:57.071117 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:57.074179 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:57.074934 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:57.075077 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:57.081084 (+ 6007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:57.081093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:57.081099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:57.081162 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:57.085574 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:57.087039 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:57.087047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:57.087051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:57.087299 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:57.088045 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:57.088052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:57.088055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:57.088465 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:57.088618 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:57.089861 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0811 03:06:57.090169 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37045,"thread_start_us":40097,"threads_started":1}
I20260811 03:06:57.093788 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:57.027302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43839 (local address 127.27.109.1:51775)
0811 03:06:57.057409 (+ 30107us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:57.057498 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:57.060408 (+ 2910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:57.065577 (+ 5169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:57.065593 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:57.066406 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:06:57.067482 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:57.067516 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:57.071425 (+ 3909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:57.071431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:57.073943 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:57.073964 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:57.075527 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:57.075538 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:57.075969 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:57.076945 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:06:57.076996 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:06:57.080705 (+ 3709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:06:57.085869 (+ 5164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:57.085879 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:57.085903 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:57.086818 (+ 915us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:57.087498 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:57.087503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:57.087507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:57.087861 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:57.088777 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:06:57.088787 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:06:57.089346 (+ 559us) client_negotiation.cc:769] Sending connection context
0811 03:06:57.091339 (+ 1993us) client_negotiation.cc:240] Negotiation successful
0811 03:06:57.092019 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29625,"thread_start_us":33574,"threads_started":1}
Aug 11 03:06:57 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417616, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:06:57.096880 8765 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43839
I20260811 03:06:57.097251 8765 heartbeater.cc:464] Registering TS with master...
I20260811 03:06:57.098505 8765 heartbeater.cc:511] Master 127.27.109.62:43839 requested a full tablet report, sending...
I20260811 03:06:57.101547 8535 ts_manager.cc:194] Registered new tserver with Master: 529e475299d64c54926679f795219754 (127.27.109.1:34049)
I20260811 03:06:57.104457 8535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:51775
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
W20260811 03:06:57.164388 28084 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.109.2: real 0.133s user 0.000s sys 0.027s
I20260811 03:06:57.165387 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:53375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43839
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:06:57 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417616, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
W20260811 03:06:57.751098 8772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:06:57.751557 8772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:06:57.751807 8772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:06:57.764333 8772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:06:57.764639 8772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:06:57.764918 8772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:06:57.781996 8772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.109.20:53375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:06:57.784104 8772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:06:57.785956 8772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:06:57.803207 8787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:57.803318 8788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:57.803366 8790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:06:57.805536 8772 server_base.cc:1034] running on GCE node
I20260811 03:06:57.806308 8772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:06:57.807529 8772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:06:57.808761 8772 hybrid_clock.cc:648] HybridClock initialized: now 1786417617808719 us; error 36 us; skew 500 ppm
I20260811 03:06:57.809159 8772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:06:57 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417617, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:06:57.812762 8772 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:06:57.815022 8772 webserver.cc:460] Webserver started at http://127.27.109.2:36115/ using document root <none> and password file <none>
I20260811 03:06:57.815985 8772 fs_manager.cc:359] Metadata directory not provided
I20260811 03:06:57.816202 8772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:06:57.816565 8772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 03:06:57 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417617, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:06:57.833380 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "54960b48fc4a47a5baaf1c53f8d096d5"
format_stamp: "Formatted at 2026-08-11 03:06:57 on dist-test-slave-8tdl"
server_key: "2cd3572f02f05a01407176f813f49c7f"
server_key_iv: "29b5087ef458024c5ce4382206d04413"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:57.834543 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "54960b48fc4a47a5baaf1c53f8d096d5"
format_stamp: "Formatted at 2026-08-11 03:06:57 on dist-test-slave-8tdl"
server_key: "2cd3572f02f05a01407176f813f49c7f"
server_key_iv: "29b5087ef458024c5ce4382206d04413"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:57.842805 8772 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s
I20260811 03:06:57.863095 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:57.865310 8772 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260811 03:06:57.865547 8772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "54960b48fc4a47a5baaf1c53f8d096d5"
format_stamp: "Formatted at 2026-08-11 03:06:57 on dist-test-slave-8tdl"
server_key: "2cd3572f02f05a01407176f813f49c7f"
server_key_iv: "29b5087ef458024c5ce4382206d04413"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:57.865991 8772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:06:57.909401 8772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:06:57.915508 8772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:06:57.915969 8772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:06:57.917568 8772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:06:57.920279 8772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:06:57.920535 8772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:57.920778 8772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:06:57.920930 8772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:58.038700 8772 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:44635
I20260811 03:06:58.038916 8920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:44635 every 8 connection(s)
I20260811 03:06:58.040409 8772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:06:58.041815 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 8772
I20260811 03:06:58.042075 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:06:58.042659 28084 external_mini_cluster.cc:1428] Setting key 06f97d0528da702b6a5b5cd239deb655
Aug 11 03:06:58 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417617, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:06:58.075795 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:58.044409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:56465 (local address 127.27.109.62:43839)
0811 03:06:58.045371 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:58.045377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:58.047427 (+ 2050us) server_negotiation.cc:407] Connection header received
0811 03:06:58.051825 (+ 4398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:58.051830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:58.051981 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:58.052094 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:58.055119 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:58.056286 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:58.059807 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:58.060813 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:58.060985 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:58.065727 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:58.065738 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:58.065746 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:58.065798 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:58.071015 (+ 5217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:58.072183 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:58.072192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:58.072198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:58.072370 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:58.073116 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:58.073125 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:58.073131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:58.073563 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:58.073796 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:58.074643 (+ 847us) server_negotiation.cc:299] Negotiation successful
0811 03:06:58.075074 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":620,"threads_started":1}
I20260811 03:06:58.076620 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:58.045646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43839 (local address 127.27.109.2:56465)
0811 03:06:58.047225 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:58.047290 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:58.051341 (+ 4051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:58.052467 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:58.052483 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:58.053594 (+ 1111us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:06:58.054874 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:58.054911 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:58.056535 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:58.056543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:58.059562 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:58.059621 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:58.061191 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:58.061198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:58.061507 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:58.062427 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:06:58.062452 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:06:58.065400 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:06:58.071241 (+ 5841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:58.071250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:58.071267 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:58.072018 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:58.072514 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:58.072518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:58.072520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:58.072949 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:58.073706 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:06:58.073712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:06:58.074176 (+ 464us) client_negotiation.cc:769] Sending connection context
0811 03:06:58.074499 (+ 323us) client_negotiation.cc:240] Negotiation successful
0811 03:06:58.075159 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":735,"threads_started":1}
I20260811 03:06:58.078591 8923 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43839
I20260811 03:06:58.079118 8923 heartbeater.cc:464] Registering TS with master...
I20260811 03:06:58.080586 8923 heartbeater.cc:511] Master 127.27.109.62:43839 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
I20260811 03:06:58.082927 8535 ts_manager.cc:194] Registered new tserver with Master: 54960b48fc4a47a5baaf1c53f8d096d5 (127.27.109.2:44635)
I20260811 03:06:58.083912 8535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:56465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
I20260811 03:06:58.109124 8765 heartbeater.cc:503] Master 127.27.109.62:43839 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:06:58.122336 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:53375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43839
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:06:58.741456 8929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:06:58.741792 8929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:06:58.742066 8929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:06:58.753098 8929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:06:58.753366 8929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:06:58.753690 8929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:06:58.767330 8929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.109.20:53375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:43839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:06:58.769026 8929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:06:58.770524 8929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:06:58.791635 8935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:58.791635 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:06:58.791914 8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:06:58.793898 8929 server_base.cc:1034] running on GCE node
I20260811 03:06:58.794901 8929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:06:58.796151 8929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:06:58.797452 8929 hybrid_clock.cc:648] HybridClock initialized: now 1786417618797355 us; error 76 us; skew 500 ppm
I20260811 03:06:58.797894 8929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:06:58 dist-test-slave-8tdl krb5kdc[7999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417618, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:06:58.803017 8929 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:06:58.805750 8929 webserver.cc:460] Webserver started at http://127.27.109.3:39535/ using document root <none> and password file <none>
I20260811 03:06:58.806813 8929 fs_manager.cc:359] Metadata directory not provided
I20260811 03:06:58.807005 8929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:06:58.807487 8929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 03:06:58 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417618, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:06:58.829069 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "24e193a7a14e41f287489a33939852c2"
format_stamp: "Formatted at 2026-08-11 03:06:58 on dist-test-slave-8tdl"
server_key: "3fdb6024c532e0882d40131b304ec44e"
server_key_iv: "6d23e38bf67d50134335568963e961fb"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:58.830549 8929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "24e193a7a14e41f287489a33939852c2"
format_stamp: "Formatted at 2026-08-11 03:06:58 on dist-test-slave-8tdl"
server_key: "3fdb6024c532e0882d40131b304ec44e"
server_key_iv: "6d23e38bf67d50134335568963e961fb"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:58.840075 8929 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.002s
I20260811 03:06:58.857383 8945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:58.859191 8929 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 03:06:58.859457 8929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "24e193a7a14e41f287489a33939852c2"
format_stamp: "Formatted at 2026-08-11 03:06:58 on dist-test-slave-8tdl"
server_key: "3fdb6024c532e0882d40131b304ec44e"
server_key_iv: "6d23e38bf67d50134335568963e961fb"
server_key_version: "kuduclusterkey@0"
I20260811 03:06:58.859706 8929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:06:58.877573 8929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:06:58.883571 8929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:06:58.884106 8929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:06:58.886466 8929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:06:58.889256 8929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:06:58.889348 8929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:58.889501 8929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:06:58.889567 8929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:06:58.977226 8929 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:33161
I20260811 03:06:58.977454 9071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:33161 every 8 connection(s)
I20260811 03:06:58.979573 8929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:06:58.981971 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 8929
I20260811 03:06:58.982267 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:06:58.983017 28084 external_mini_cluster.cc:1428] Setting key 15f14a0eef18caa2076a39311a64ee64
Aug 11 03:06:58 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417618, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:06:59.012333 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:58.984509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:46109 (local address 127.27.109.62:43839)
0811 03:06:58.985468 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:58.985473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:58.990341 (+ 4868us) server_negotiation.cc:407] Connection header received
0811 03:06:58.990409 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:58.990412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:58.990487 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:58.990581 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:58.992724 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:58.993764 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:58.995788 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:58.996536 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:58.996687 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.001621 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:59.001628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:59.001632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:59.001675 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:59.006564 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.008061 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.008066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.008069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.008234 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.009167 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.009173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.009175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.009566 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:59.009744 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.011325 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.011664 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":637,"threads_started":1}
I20260811 03:06:59.013216 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:58.985611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:43839 (local address 127.27.109.3:46109)
0811 03:06:58.987148 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:58.987202 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:58.989939 (+ 2737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:58.990990 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:58.991001 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:58.991646 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:06:58.992471 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:58.992494 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:58.993959 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:58.993963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:58.995588 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:58.995609 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:58.996849 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:58.996860 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:58.997149 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:58.997952 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:06:58.997978 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:06:59.001287 (+ 3309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:06:59.006865 (+ 5578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:59.006875 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:59.006894 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:59.007794 (+ 900us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:59.008467 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:06:59.008472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:06:59.008475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:06:59.008947 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:06:59.009912 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:06:59.009920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:06:59.010361 (+ 441us) client_negotiation.cc:769] Sending connection context
0811 03:06:59.010849 (+ 488us) client_negotiation.cc:240] Negotiation successful
0811 03:06:59.011480 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"thread_start_us":644,"threads_started":1}
I20260811 03:06:59.015626 9072 heartbeater.cc:347] Connected to a master server at 127.27.109.62:43839
I20260811 03:06:59.016167 9072 heartbeater.cc:464] Registering TS with master...
I20260811 03:06:59.017400 9072 heartbeater.cc:511] Master 127.27.109.62:43839 requested a full tablet report, sending...
I20260811 03:06:59.021176 8535 ts_manager.cc:194] Registered new tserver with Master: 24e193a7a14e41f287489a33939852c2 (127.27.109.3:33161)
I20260811 03:06:59.022147 8535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:46109
I20260811 03:06:59.028155 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:06:59.056703 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.035127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.27.109.62:43839)
0811 03:06:59.035435 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.035446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.036212 (+ 766us) server_negotiation.cc:407] Connection header received
0811 03:06:59.036508 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.036514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.036671 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.036795 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:59.039476 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.041269 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.043303 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.044400 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.044579 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.048381 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:59.048390 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:59.048393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:59.048438 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:59.052819 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.053992 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.053997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.054000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.054206 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.054894 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.054899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.054902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.055251 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:59.055576 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.055901 (+ 325us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.056213 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260811 03:06:59.062387 8535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:06:59.074397 8535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:06:59.088122 8923 heartbeater.cc:503] Master 127.27.109.62:43839 was elected leader, sending a full tablet report...
I20260811 03:06:59.121973 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.092970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.27.109.3:33161)
0811 03:06:59.094664 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.094708 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.100033 (+ 5325us) server_negotiation.cc:407] Connection header received
0811 03:06:59.100220 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.100233 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.100740 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.101078 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:59.105360 (+ 4282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.111348 (+ 5988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.117056 (+ 5708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.119315 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.119474 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.119792 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.120860 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.121077 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1387,"thread_start_us":958,"threads_started":1}
I20260811 03:06:59.122295 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.092763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:33161 (local address 127.0.0.1:37346)
0811 03:06:59.095769 (+ 3006us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:59.095828 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:59.096372 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:59.101765 (+ 5393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:59.101778 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:59.101832 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:59.104540 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.104572 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.111593 (+ 7021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.111608 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.116826 (+ 5218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.116840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.120101 (+ 3261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.120108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.120530 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.120585 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 03:06:59.120783 (+ 198us) client_negotiation.cc:240] Negotiation successful
0811 03:06:59.121022 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2376,"mutex_wait_us":27,"thread_start_us":1854,"threads_started":1}
I20260811 03:06:59.127043 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.094120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.27.109.1:34049)
0811 03:06:59.111041 (+ 16921us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.111054 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.111101 (+ 47us) server_negotiation.cc:407] Connection header received
0811 03:06:59.111260 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.111272 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.112255 (+ 983us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.112561 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:59.115410 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.117813 (+ 2403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.123137 (+ 5324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.125085 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.125306 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.125699 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.126101 (+ 402us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.126368 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16582,"thread_start_us":16277,"threads_started":1}
I20260811 03:06:59.127086 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.093042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34049 (local address 127.0.0.1:53822)
0811 03:06:59.109098 (+ 16056us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:59.109136 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:59.109417 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:59.112994 (+ 3577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:59.112999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:59.113035 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:59.114920 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.115114 (+ 194us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.118050 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.118055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.122888 (+ 4833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.122909 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.125402 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.125411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.125798 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.125844 (+ 46us) client_negotiation.cc:769] Sending connection context
0811 03:06:59.125992 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 03:06:59.126173 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15716,"mutex_wait_us":3,"thread_start_us":19629,"threads_started":1}
I20260811 03:06:59.128502 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.096626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.27.109.2:44635)
0811 03:06:59.110386 (+ 13760us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.110394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.113556 (+ 3162us) server_negotiation.cc:407] Connection header received
0811 03:06:59.115456 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.115479 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.115792 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.116116 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:59.117536 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.119822 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.124623 (+ 4801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.126670 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.126870 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.127214 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.127884 (+ 670us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.128068 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13498,"thread_start_us":13308,"threads_started":1}
I20260811 03:06:59.128571 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.096207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44635 (local address 127.0.0.1:36296)
0811 03:06:59.113296 (+ 17089us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:59.113317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:59.113505 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:59.116139 (+ 2634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:59.116145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:59.116192 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:59.117297 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.117314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.120089 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.120096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.124325 (+ 4229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.124345 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.127491 (+ 3146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.127498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.127679 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.127717 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 03:06:59.127836 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 03:06:59.127981 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16793,"thread_start_us":13327,"threads_started":1}
I20260811 03:06:59.128508 9001 tablet_service.cc:1467] Processing CreateTablet for tablet 431f5c5fef454d088f17d7988dd01a9e (DEFAULT_TABLE table=test-table [id=3081cb9f35c14dd1bd57cf175e8750d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:06:59.131246 9001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f5c5fef454d088f17d7988dd01a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:06:59.134506 8694 tablet_service.cc:1467] Processing CreateTablet for tablet 431f5c5fef454d088f17d7988dd01a9e (DEFAULT_TABLE table=test-table [id=3081cb9f35c14dd1bd57cf175e8750d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:06:59.134246 8850 tablet_service.cc:1467] Processing CreateTablet for tablet 431f5c5fef454d088f17d7988dd01a9e (DEFAULT_TABLE table=test-table [id=3081cb9f35c14dd1bd57cf175e8750d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:06:59.136457 8694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f5c5fef454d088f17d7988dd01a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:06:59.136530 8850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f5c5fef454d088f17d7988dd01a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:06:59.157311 9093 tablet_bootstrap.cc:492] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Bootstrap starting.
I20260811 03:06:59.162089 9094 tablet_bootstrap.cc:492] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Bootstrap starting.
I20260811 03:06:59.163864 9095 tablet_bootstrap.cc:492] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Bootstrap starting.
I20260811 03:06:59.166308 9093 tablet_bootstrap.cc:654] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Neither blocks nor log segments found. Creating new log.
I20260811 03:06:59.169031 9094 tablet_bootstrap.cc:654] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Neither blocks nor log segments found. Creating new log.
I20260811 03:06:59.169135 9093 log.cc:824] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Log is configured to *not* fsync() on all Append() calls
I20260811 03:06:59.170733 9095 tablet_bootstrap.cc:654] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Neither blocks nor log segments found. Creating new log.
I20260811 03:06:59.171314 9094 log.cc:824] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Log is configured to *not* fsync() on all Append() calls
I20260811 03:06:59.172660 9095 log.cc:824] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Log is configured to *not* fsync() on all Append() calls
I20260811 03:06:59.174973 9093 tablet_bootstrap.cc:492] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: No bootstrap required, opened a new log
I20260811 03:06:59.175405 9093 ts_tablet_manager.cc:1397] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260811 03:06:59.176535 9094 tablet_bootstrap.cc:492] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: No bootstrap required, opened a new log
I20260811 03:06:59.176937 9094 ts_tablet_manager.cc:1397] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260811 03:06:59.178735 9095 tablet_bootstrap.cc:492] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: No bootstrap required, opened a new log
I20260811 03:06:59.179138 9095 ts_tablet_manager.cc:1397] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20260811 03:06:59.189648 9094 raft_consensus.cc:348] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.190516 9094 raft_consensus.cc:374] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:06:59.190769 9094 raft_consensus.cc:729] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e193a7a14e41f287489a33939852c2, State: Initialized, Role: FOLLOWER
I20260811 03:06:59.191701 9094 consensus_queue.cc:260] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.192836 9093 raft_consensus.cc:348] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.193802 9072 heartbeater.cc:503] Master 127.27.109.62:43839 was elected leader, sending a full tablet report...
I20260811 03:06:59.193856 9093 raft_consensus.cc:374] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:06:59.194077 9093 raft_consensus.cc:729] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529e475299d64c54926679f795219754, State: Initialized, Role: FOLLOWER
I20260811 03:06:59.194765 9094 ts_tablet_manager.cc:1428] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260811 03:06:59.195785 9093 consensus_queue.cc:260] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.196578 9095 raft_consensus.cc:348] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.197322 9095 raft_consensus.cc:374] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:06:59.197521 9095 raft_consensus.cc:729] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54960b48fc4a47a5baaf1c53f8d096d5, State: Initialized, Role: FOLLOWER
I20260811 03:06:59.198652 9093 ts_tablet_manager.cc:1428] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Time spent starting tablet: real 0.023s user 0.021s sys 0.001s
I20260811 03:06:59.198462 9095 consensus_queue.cc:260] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.201766 9095 ts_tablet_manager.cc:1428] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Time spent starting tablet: real 0.022s user 0.016s sys 0.006s
W20260811 03:06:59.233232 9073 tablet.cc:2369] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:06:59.294252 8924 tablet.cc:2369] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:06:59.317474 8767 tablet.cc:2369] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:06:59.386722 9100 raft_consensus.cc:482] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:06:59.387334 9100 raft_consensus.cc:504] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.389773 9100 leader_election.cc:288] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 529e475299d64c54926679f795219754 (127.27.109.1:34049), 54960b48fc4a47a5baaf1c53f8d096d5 (127.27.109.2:44635)
I20260811 03:06:59.397814 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.390348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:38297 (local address 127.27.109.2:44635)
0811 03:06:59.390635 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.390642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.390935 (+ 293us) server_negotiation.cc:407] Connection header received
0811 03:06:59.391183 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.391188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.391248 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.391350 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:59.391912 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.393434 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.395800 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.396769 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.396950 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.397040 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.397334 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.397474 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 03:06:59.397812 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.390459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44635 (local address 127.27.109.3:38297)
0811 03:06:59.390815 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:59.390832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:59.391015 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:59.391389 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:59.391394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:59.391422 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:59.391755 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.391766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.393655 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.393659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.395636 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.395650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.396948 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.396954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.397159 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.397184 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 03:06:59.397308 (+ 124us) client_negotiation.cc:240] Negotiation successful
0811 03:06:59.397460 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260811 03:06:59.399473 8870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f5c5fef454d088f17d7988dd01a9e" candidate_uuid: "24e193a7a14e41f287489a33939852c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" is_pre_election: true
I20260811 03:06:59.399935 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.390527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34049 (local address 127.27.109.3:35719)
0811 03:06:59.391955 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0811 03:06:59.391971 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 03:06:59.392121 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:06:59.392597 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:06:59.392606 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:06:59.392639 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:06:59.393411 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.393423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.394993 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.395002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.397007 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:06:59.397021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.399079 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.399084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:06:59.399257 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.399289 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 03:06:59.399369 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 03:06:59.399470 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"thread_start_us":1037,"threads_started":1}
I20260811 03:06:59.400058 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.390401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:35719 (local address 127.27.109.1:34049)
0811 03:06:59.390635 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.390642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.392089 (+ 1447us) server_negotiation.cc:407] Connection header received
0811 03:06:59.392277 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.392282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.392360 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.392492 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:06:59.393676 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.394824 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.397198 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.398784 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.399055 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.399152 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.399480 (+ 328us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.399640 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 03:06:59.400630 8870 raft_consensus.cc:2454] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e193a7a14e41f287489a33939852c2 in term 0.
I20260811 03:06:59.401718 8947 leader_election.cc:302] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e193a7a14e41f287489a33939852c2, 54960b48fc4a47a5baaf1c53f8d096d5; no voters:
I20260811 03:06:59.401469 8716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f5c5fef454d088f17d7988dd01a9e" candidate_uuid: "24e193a7a14e41f287489a33939852c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e475299d64c54926679f795219754" is_pre_election: true
I20260811 03:06:59.402472 9100 raft_consensus.cc:2781] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:06:59.402400 8716 raft_consensus.cc:2454] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24e193a7a14e41f287489a33939852c2 in term 0.
I20260811 03:06:59.402771 9100 raft_consensus.cc:482] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:06:59.402911 9100 raft_consensus.cc:3037] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:06:59.407008 9100 raft_consensus.cc:504] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.407915 9100 leader_election.cc:288] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [CANDIDATE]: Term 1 election: Requested vote from peers 529e475299d64c54926679f795219754 (127.27.109.1:34049), 54960b48fc4a47a5baaf1c53f8d096d5 (127.27.109.2:44635)
I20260811 03:06:59.408602 8716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f5c5fef454d088f17d7988dd01a9e" candidate_uuid: "24e193a7a14e41f287489a33939852c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "529e475299d64c54926679f795219754"
I20260811 03:06:59.408697 8870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f5c5fef454d088f17d7988dd01a9e" candidate_uuid: "24e193a7a14e41f287489a33939852c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54960b48fc4a47a5baaf1c53f8d096d5"
I20260811 03:06:59.408860 8716 raft_consensus.cc:3037] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:06:59.409010 8870 raft_consensus.cc:3037] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:06:59.412158 8716 raft_consensus.cc:2454] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e193a7a14e41f287489a33939852c2 in term 1.
I20260811 03:06:59.412568 8870 raft_consensus.cc:2454] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24e193a7a14e41f287489a33939852c2 in term 1.
I20260811 03:06:59.412772 8950 leader_election.cc:302] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24e193a7a14e41f287489a33939852c2, 529e475299d64c54926679f795219754; no voters:
I20260811 03:06:59.413437 9100 raft_consensus.cc:2781] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:06:59.414426 9100 raft_consensus.cc:686] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 1 LEADER]: Becoming Leader. State: Replica: 24e193a7a14e41f287489a33939852c2, State: Running, Role: LEADER
I20260811 03:06:59.415151 9100 consensus_queue.cc:237] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } }
I20260811 03:06:59.422113 8533 catalog_manager.cc:5495] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24e193a7a14e41f287489a33939852c2 (127.27.109.3). New cstate: current_term: 1 leader_uuid: "24e193a7a14e41f287489a33939852c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24e193a7a14e41f287489a33939852c2" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 33161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 } health_report { overall_health: UNKNOWN } } }
I20260811 03:06:59.513856 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.506040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.27.109.3:33161)
0811 03:06:59.506351 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.506356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.506535 (+ 179us) server_negotiation.cc:407] Connection header received
0811 03:06:59.506733 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.506737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.506825 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.506963 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:06:59.507488 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.508516 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.510869 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.511612 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.511727 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.512516 (+ 789us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:06:59.513091 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:06:59.513214 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.513434 (+ 220us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.513526 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 03:06:59.525147 8870 raft_consensus.cc:1261] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 1 FOLLOWER]: Refusing update from remote peer 24e193a7a14e41f287489a33939852c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:06:59.525187 8716 raft_consensus.cc:1261] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 1 FOLLOWER]: Refusing update from remote peer 24e193a7a14e41f287489a33939852c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:06:59.526357 9106 consensus_queue.cc:1035] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54960b48fc4a47a5baaf1c53f8d096d5" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:06:59.527055 9100 consensus_queue.cc:1035] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "529e475299d64c54926679f795219754" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:06:59.568092 8533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:06:59.568425 8533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:06:59.573345 8533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b27ae54eac404602ab578dbe8c5712cf: Sending DeleteTablet for 3 replicas of tablet 431f5c5fef454d088f17d7988dd01a9e
I20260811 03:06:59.574995 9001 tablet_service.cc:1514] Processing DeleteTablet for tablet 431f5c5fef454d088f17d7988dd01a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:06:59 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:37346
I20260811 03:06:59.575193 8694 tablet_service.cc:1514] Processing DeleteTablet for tablet 431f5c5fef454d088f17d7988dd01a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:06:59 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:53822
I20260811 03:06:59.576917 8850 tablet_service.cc:1514] Processing DeleteTablet for tablet 431f5c5fef454d088f17d7988dd01a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:06:59 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36296
I20260811 03:06:59.577277 9121 tablet_replica.cc:330] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: stopping tablet replica
I20260811 03:06:59.577661 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 8620
I20260811 03:06:59.577847 9121 raft_consensus.cc:2229] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:06:59.578933 9121 raft_consensus.cc:2258] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:06:59.580329 9122 tablet_replica.cc:330] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: stopping tablet replica
I20260811 03:06:59.580821 9122 raft_consensus.cc:2229] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:06:59.581629 9122 raft_consensus.cc:2258] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:06:59.584292 9122 ts_tablet_manager.cc:1905] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:06:59.584400 9121 ts_tablet_manager.cc:1905] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:06:59.592641 9122 ts_tablet_manager.cc:1918] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:06:59.592851 9122 log.cc:1196] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/431f5c5fef454d088f17d7988dd01a9e
I20260811 03:06:59.593546 9122 ts_tablet_manager.cc:1939] T 431f5c5fef454d088f17d7988dd01a9e P 24e193a7a14e41f287489a33939852c2: Deleting consensus metadata
I20260811 03:06:59.596038 8516 catalog_manager.cc:4841] TS 24e193a7a14e41f287489a33939852c2 (127.27.109.3:33161): tablet 431f5c5fef454d088f17d7988dd01a9e (table test-table [id=3081cb9f35c14dd1bd57cf175e8750d6]) successfully deleted
I20260811 03:06:59.597247 9123 tablet_replica.cc:330] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: stopping tablet replica
I20260811 03:06:59.597820 9123 raft_consensus.cc:2229] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:06:59.598845 9123 raft_consensus.cc:2258] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:06:59.601611 9123 ts_tablet_manager.cc:1905] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:06:59.602521 9121 ts_tablet_manager.cc:1918] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 11 03:06:59 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:06:59.602880 9121 log.cc:1196] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/431f5c5fef454d088f17d7988dd01a9e
I20260811 03:06:59.603771 9121 ts_tablet_manager.cc:1939] T 431f5c5fef454d088f17d7988dd01a9e P 529e475299d64c54926679f795219754: Deleting consensus metadata
I20260811 03:06:59.606746 8520 catalog_manager.cc:4841] TS 529e475299d64c54926679f795219754 (127.27.109.1:34049): tablet 431f5c5fef454d088f17d7988dd01a9e (table test-table [id=3081cb9f35c14dd1bd57cf175e8750d6]) successfully deleted
I20260811 03:06:59.611843 9123 ts_tablet_manager.cc:1918] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:06:59.612082 9123 log.cc:1196] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/431f5c5fef454d088f17d7988dd01a9e
I20260811 03:06:59.612684 9123 ts_tablet_manager.cc:1939] T 431f5c5fef454d088f17d7988dd01a9e P 54960b48fc4a47a5baaf1c53f8d096d5: Deleting consensus metadata
I20260811 03:06:59.614353 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.580405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.27.109.1:34049)
0811 03:06:59.580582 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.580588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.580761 (+ 173us) server_negotiation.cc:407] Connection header received
0811 03:06:59.581122 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.581131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.581243 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.581339 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:59.591372 (+ 10033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.592864 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.595866 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.597104 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.597251 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.604652 (+ 7401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:59.604667 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:59.604685 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:59.604769 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:59.609961 (+ 5192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.611201 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.611209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.611211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.611403 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.611932 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.611935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.611938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.612581 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:59.612985 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.613452 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.613767 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 03:06:59.615155 8516 catalog_manager.cc:4841] TS 54960b48fc4a47a5baaf1c53f8d096d5 (127.27.109.2:44635): tablet 431f5c5fef454d088f17d7988dd01a9e (table test-table [id=3081cb9f35c14dd1bd57cf175e8750d6]) successfully deleted
I20260811 03:06:59.615581 8760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:06:59.744048 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 8620
I20260811 03:06:59.766839 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 8772
Aug 11 03:06:59 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:06:59.784665 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.767766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.27.109.2:44635)
0811 03:06:59.767962 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.767968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.767989 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:06:59.768114 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.768119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.768211 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.768317 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:59.769695 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.770413 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.772055 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.772509 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.772655 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.777901 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:59.777917 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:59.777935 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:59.777979 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:59.781473 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.782280 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.782288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.782290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.782434 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.782928 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.782939 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.782941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.783349 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:59.783662 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.784011 (+ 349us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.784286 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 03:06:59.785420 8916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:06:59.879632 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 8772
I20260811 03:06:59.903358 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 8929
Aug 11 03:06:59 dist-test-slave-8tdl krb5kdc[7999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:06:59.923911 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:06:59.904183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.27.109.3:33161)
0811 03:06:59.904367 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 03:06:59.904371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:06:59.904424 (+ 53us) server_negotiation.cc:407] Connection header received
0811 03:06:59.904671 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:06:59.904675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:06:59.904744 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:06:59.904828 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:06:59.907123 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.908031 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.910260 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:06:59.910968 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:06:59.911102 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:06:59.917252 (+ 6150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:06:59.917265 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:06:59.917279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:06:59.917334 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:06:59.920674 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.921490 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.921501 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.921504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.921642 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:06:59.922066 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:06:59.922072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:06:59.922075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:06:59.922441 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:06:59.922858 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0811 03:06:59.923289 (+ 431us) server_negotiation.cc:299] Negotiation successful
0811 03:06:59.923546 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 03:06:59.924731 9066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:00.052143 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 8929
I20260811 03:07:00.076258 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 8498
I20260811 03:07:00.077584 8566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:00.195801 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 8498
I20260811 03:07:00.225118 28084 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 03:07:00.571898 28084 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 03:07:00.572238 28084 mini_ranger.cc:66] Stopping Ranger...
I20260811 03:07:00.928791 28084 mini_ranger.cc:68] Stopped Ranger
2026-08-11 03:07:00.929 UTC [8031] LOG: received smart shutdown request
2026-08-11 03:07:00.933 UTC [8031] LOG: background worker "logical replication launcher" (PID 8040) exited with exit code 1
2026-08-11 03:07:00.933 UTC [8034] LOG: shutting down
2026-08-11 03:07:01.162 UTC [8031] LOG: database system is shut down
2026-08-11T03:07:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57094 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:07:01 dist-test-slave-8tdl krb5kdc[9141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:07:01 dist-test-slave-8tdl krb5kdc[9141](info): set up 2 sockets
Aug 11 03:07:01 dist-test-slave-8tdl krb5kdc[9141](info): commencing operation
krb5kdc: starting...
W20260811 03:07:02.555496 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:07:02 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:07:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/rangeradmin_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/rangeradmin_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/rangerlookup_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/rangerlookup_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
I20260811 03:07:03.035663 28084 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWXDOoZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/postgres -l logfile start
2026-08-11 03:07:06.021 UTC [9173] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 03:07:06.021 UTC [9173] LOG: listening on IPv4 address "127.27.109.20", port 44757
2026-08-11 03:07:06.026 UTC [9173] LOG: listening on Unix socket "/tmp/.s.PGSQL.44757"
2026-08-11 03:07:06.053 UTC [9176] LOG: database system was shut down at 2026-08-11 03:07:04 UTC
2026-08-11 03:07:06.057 UTC [9173] LOG: database system is ready to accept connections
I20260811 03:07:07.130658 28084 mini_postgres.cc:96] Postgres bound to 44757
2026-08-11 03:07:07.155 UTC [9184] FATAL: database "slave" does not exist
127.27.109.20:44757 - accepting connections
I20260811 03:07:07.156256 28084 mini_ranger.cc:161] Starting Ranger...
I20260811 03:07:07.191339 28084 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 03:07:07.779700 28084 mini_ranger.cc:87] Created ranger Postgres database
I20260811 03:07:07.779848 28084 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-admin
2026-08-11 03:07:08,097 [I] DB FLAVOR :POSTGRES
2026-08-11 03:07:08,098 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 03:07:08,098 [I] Checking connection..
2026-08-11 03:07:08,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 03:07:08,358 [I] Checking connection passed.
2026-08-11 03:07:08,359 [I] --------- Verifying version history table ---------
2026-08-11 03:07:08,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 03:07:08,643 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 03:07:08,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 03:07:08,913 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 03:07:08,913 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 03:07:08,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 03:07:09.184 UTC [9254] WARNING: there is no transaction in progress
2026-08-11 03:07:09,192 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 03:07:09,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 03:07:09,465 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 03:07:09,465 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 03:07:09,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 03:07:09,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 03:07:09,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8tdl.c.gcp-upstream.internal','N') ;"
2026-08-11 03:07:10,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 03:07:10,519 [I] Table x_portal_user does not exist in database ranger
2026-08-11 03:07:10,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 03:07:10,800 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 03:07:10,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 03:07:11,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 03:07:11,350 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 03:07:11,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 03:07:12.477 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.496 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.516 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.535 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.557 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.578 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.643 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.651 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.668 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.680 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.691 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.707 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.718 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.726 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.734 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:12.742 UTC [9369] WARNING: there is no transaction in progress
2026-08-11 03:07:13,538 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 03:07:13,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 03:07:13,812 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 03:07:13,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 03:07:14,097 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 03:07:14,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 03:07:14,367 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 03:07:14,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 03:07:14,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 03:07:14,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8tdl.c.gcp-upstream.internal';"
2026-08-11 03:07:15,180 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 03:07:15,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 03:07:15,443 [I] DB_PATCHES have already been applied
I20260811 03:07:15.450165 28084 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWXDOoZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 03:07:15.450268 28084 mini_ranger.cc:193] Using host: 127.27.109.20
I20260811 03:07:15.473352 28084 mini_ranger.cc:227] Ranger admin URL: http://127.27.109.20:43189
Aug 11, 2026 3:07:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 3:07:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 3:07:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 3:07:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 3:07:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 3:07:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 3:07:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 03:07:16.730993 28084 mini_ranger.cc:160] Time spent starting Ranger: real 9.575s user 0.008s sys 0.060s
Aug 11, 2026 3:07:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 3:07:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 3:07:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 3:07:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 3:07:39 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 03:07:39.635676 28084 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.109.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.109.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/rangerkms_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.109.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/rangerkms_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.109.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.109.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.109.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/HTTP_127.27.109.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 03:07:39 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417659, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 03:07:39.894796 28084 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 03:07:39.924552 28084 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 03:07:41.015702 28084 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 03:07:41.015836 28084 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms
I20260811 03:07:41.015869 28084 mini_ranger_kms.cc:227] Using postgres at 127.27.109.20:44757
2026-08-11 03:07:41,446 [I] DB FLAVOR :POSTGRES
2026-08-11 03:07:41,447 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 03:07:41,447 [I] Checking connection
2026-08-11 03:07:41,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 03:07:41,703 [I] connection success
2026-08-11 03:07:41,703 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 03:07:41,703 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 03:07:41,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 03:07:41,990 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 03:07:41,991 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 03:07:41,991 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 03:07:41,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.109.20:44757/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 03:07:42,312 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 03:07:42.865777 28084 mini_ranger_kms.cc:310] Created kms service
I20260811 03:07:43.102959 28084 mini_ranger_kms.cc:326] Created kudu user
I20260811 03:07:43.151037 28084 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 03:07:43 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417659, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:43.632299 28084 mini_ranger_kms.cc:384] Added ranger policy
I20260811 03:07:43.632673 28084 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWXDOoZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWXDOoZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 03:07:43.632727 28084 mini_ranger_kms.cc:240] Using host: 127.27.109.20
I20260811 03:07:43.655444 28084 mini_ranger_kms.cc:276] Ranger KMS PID: 9565
I20260811 03:07:43.655602 28084 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.109.20:50715
Aug 11, 2026 3:07:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 3:07:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 3:07:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 3:07:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 3:07:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 3:07:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 03:07:46 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786417666, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:46.147274 28084 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.252s user 0.092s sys 0.375s
I20260811 03:07:46.147544 28084 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 03:07:46.147732 28084 mini_ranger_kms.cc:401] 127.27.109.20:50715/kms/v1/keys
Aug 11 03:07:46 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417666, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 11 03:07:46 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417666, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.109.20@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 11 03:07:49 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:07:49.523603 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41493
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41493
--ranger_config_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:50715/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:07:49.892088 9618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:07:49.892421 9618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:07:49.892514 9618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:07:49.900141 9618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:07:49.900278 9618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:07:49.900323 9618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:07:49.900373 9618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:07:49.900422 9618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:07:49.911118 9618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.109.20:50715/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41493
--ranger_config_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:07:49.912510 9618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:07:49.913892 9618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:07:49.928059 9628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:49.928010 9625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:49.928107 9624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:07:49.929250 9618 server_base.cc:1034] running on GCE node
I20260811 03:07:49.930318 9618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:07:49.932296 9618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:07:49.933602 9618 hybrid_clock.cc:648] HybridClock initialized: now 1786417669933568 us; error 53 us; skew 500 ppm
I20260811 03:07:49.934150 9618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:07:49 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417669, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:49.943173 9618 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:07:49.945134 9618 webserver.cc:460] Webserver started at http://127.27.109.62:43431/ using document root <none> and password file <none>
I20260811 03:07:49.945945 9618 fs_manager.cc:359] Metadata directory not provided
I20260811 03:07:49.946113 9618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:07:49.946483 9618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 03:07:49 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417669, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:50.021827 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "515de020538e4359942e2092705e3a8f"
format_stamp: "Formatted at 2026-08-11 03:07:50 on dist-test-slave-8tdl"
server_key: "fe4ef223e351f4b737edec3d6d473ed1"
server_key_iv: "bed71e9f1797723d462a883fbfd46d8d"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:50.023262 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "515de020538e4359942e2092705e3a8f"
format_stamp: "Formatted at 2026-08-11 03:07:50 on dist-test-slave-8tdl"
server_key: "fe4ef223e351f4b737edec3d6d473ed1"
server_key_iv: "bed71e9f1797723d462a883fbfd46d8d"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:50.029567 9618 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260811 03:07:50.051432 9635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:50.053434 9618 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260811 03:07:50.053673 9618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "515de020538e4359942e2092705e3a8f"
format_stamp: "Formatted at 2026-08-11 03:07:50 on dist-test-slave-8tdl"
server_key: "fe4ef223e351f4b737edec3d6d473ed1"
server_key_iv: "bed71e9f1797723d462a883fbfd46d8d"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:50.054016 9618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:07:50.076009 9618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:07:50.080786 9618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:07:50.081168 9618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:07:50.102936 9618 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41493
I20260811 03:07:50.102955 9689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41493 every 8 connection(s)
I20260811 03:07:50.104489 9618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:07:50.108567 9690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:07:50.109233 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 9618
I20260811 03:07:50.109450 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:07:50.110384 28084 external_mini_cluster.cc:1428] Setting key d464d809c97bde9d1dc7c617476d14fb
I20260811 03:07:50.124626 9690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Bootstrap starting.
Aug 11 03:07:50 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:07:50.128618 9690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Neither blocks nor log segments found. Creating new log.
I20260811 03:07:50.130223 9690 log.cc:824] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Log is configured to *not* fsync() on all Append() calls
I20260811 03:07:50.133939 9690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: No bootstrap required, opened a new log
I20260811 03:07:50.137105 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:50.113947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.27.109.62:41493)
0811 03:07:50.115296 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:50.115312 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:50.115384 (+ 72us) server_negotiation.cc:407] Connection header received
0811 03:07:50.117187 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:50.117213 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:50.117713 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:50.118168 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:50.119696 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:50.121132 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:50.122577 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:50.123423 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:50.123649 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:50.128118 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:50.128131 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:50.128154 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:50.128213 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:50.132299 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:50.133189 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:50.133198 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:50.133205 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:50.133383 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:50.133845 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:50.133850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:50.133856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:50.134579 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:50.135013 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:50.135509 (+ 496us) server_negotiation.cc:299] Negotiation successful
0811 03:07:50.135966 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":799,"threads_started":1}
I20260811 03:07:50.143711 9690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } }
I20260811 03:07:50.144316 9690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:07:50.144446 9690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515de020538e4359942e2092705e3a8f, State: Initialized, Role: FOLLOWER
I20260811 03:07:50.145128 9690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } }
I20260811 03:07:50.145385 9690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:07:50.145521 9690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:07:50.145718 9690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:07:50.148221 9690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } }
I20260811 03:07:50.148809 9690 leader_election.cc:302] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515de020538e4359942e2092705e3a8f; no voters:
I20260811 03:07:50.149463 9690 leader_election.cc:288] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:07:50.149632 9698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:07:50.150424 9698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 1 LEADER]: Becoming Leader. State: Replica: 515de020538e4359942e2092705e3a8f, State: Running, Role: LEADER
I20260811 03:07:50.151074 9698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } }
I20260811 03:07:50.151969 9690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:07:50.154971 9699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515de020538e4359942e2092705e3a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } } }
I20260811 03:07:50.155001 9700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515de020538e4359942e2092705e3a8f. Latest consensus state: current_term: 1 leader_uuid: "515de020538e4359942e2092705e3a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } } }
I20260811 03:07:50.155557 9699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: This master's current role is: LEADER
I20260811 03:07:50.156337 9700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: This master's current role is: LEADER
I20260811 03:07:50.157334 9708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:07:50.163098 9708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:07:50.163955 9690 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 03:07:50.185812 9708 catalog_manager.cc:1347] Generated new cluster ID: 988a7e1f273543159d8e2b217a3f788a
I20260811 03:07:50.185997 9708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:07:50.221467 9708 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:07:50.223693 9708 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:07:50.233901 9708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Generated new TSK 0
I20260811 03:07:50.235225 9708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:07:51.115658 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:50715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41493
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:07:51 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786417671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 03:07:51.603416 9746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:07:51.604033 9746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:07:51.604254 9746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:07:51.613538 9746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:07:51.613729 9746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:07:51.613932 9746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:07:51.626346 9746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.109.20:50715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:07:51.628070 9746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:07:51.629513 9746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:07:51.651820 9760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:51.651829 9758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:51.655782 9757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:07:51.672389 9746 server_base.cc:1034] running on GCE node
I20260811 03:07:51.673480 9746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:07:51.679647 9746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:07:51.681133 9746 hybrid_clock.cc:648] HybridClock initialized: now 1786417671681010 us; error 122 us; skew 500 ppm
I20260811 03:07:51.681634 9746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:07:51 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:51.686494 9746 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:07:51.688861 9746 webserver.cc:460] Webserver started at http://127.27.109.1:40583/ using document root <none> and password file <none>
I20260811 03:07:51.690047 9746 fs_manager.cc:359] Metadata directory not provided
I20260811 03:07:51.690305 9746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:07:51.690826 9746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 03:07:51 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:51.717783 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "e35523f49d5745acbaaae245e804d606"
format_stamp: "Formatted at 2026-08-11 03:07:51 on dist-test-slave-8tdl"
server_key: "c3bd091e514b3425956d649b4313e10d"
server_key_iv: "2dfaca46aeb019aa90c46bc509cf4dae"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:51.722393 9746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "e35523f49d5745acbaaae245e804d606"
format_stamp: "Formatted at 2026-08-11 03:07:51 on dist-test-slave-8tdl"
server_key: "c3bd091e514b3425956d649b4313e10d"
server_key_iv: "2dfaca46aeb019aa90c46bc509cf4dae"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:51.744786 9746 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.007s sys 0.006s
I20260811 03:07:51.783046 9767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:51.785697 9746 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.008s sys 0.000s
I20260811 03:07:51.786082 9746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "e35523f49d5745acbaaae245e804d606"
format_stamp: "Formatted at 2026-08-11 03:07:51 on dist-test-slave-8tdl"
server_key: "c3bd091e514b3425956d649b4313e10d"
server_key_iv: "2dfaca46aeb019aa90c46bc509cf4dae"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:51.786482 9746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:07:51.844374 9746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:07:51.850812 9746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:07:51.851280 9746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:07:51.871275 9746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:07:51.879904 9746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:07:51.880074 9746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:51.880270 9746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:07:51.880367 9746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 03:07:51 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:52.027815 9746 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:33467
I20260811 03:07:52.027876 9893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:33467 every 8 connection(s)
I20260811 03:07:52.029778 9746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:07:52.033071 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 9746
I20260811 03:07:52.033280 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:07:52.034129 28084 external_mini_cluster.cc:1428] Setting key e99723347b611e0fbf474eb16939cb27
Aug 11 03:07:52 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:07:52.063813 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:52.033046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:39513 (local address 127.27.109.62:41493)
0811 03:07:52.043834 (+ 10788us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:52.043842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:52.043869 (+ 27us) server_negotiation.cc:407] Connection header received
0811 03:07:52.043954 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:52.043960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:52.044067 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:52.046360 (+ 2293us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:52.046659 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:52.047495 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:52.049156 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:52.049586 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:52.049728 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:52.054665 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:52.054696 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:52.054704 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:52.054753 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:52.059044 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:52.060335 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:52.060343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:52.060350 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:52.060543 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:52.061223 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:52.061231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:52.061238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:52.061665 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:52.061800 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:52.063002 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0811 03:07:52.063289 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10619,"thread_start_us":11134,"threads_started":1}
I20260811 03:07:52.066826 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:52.033862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41493 (local address 127.27.109.1:39513)
0811 03:07:52.036421 (+ 2559us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:52.036477 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:52.039194 (+ 2717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:52.044719 (+ 5525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:52.044736 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:52.045476 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:07:52.046422 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:52.046464 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:52.047676 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:52.047683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:52.048987 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:52.049008 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:52.049833 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:52.049844 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:52.050206 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:52.051147 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:07:52.051185 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:07:52.054360 (+ 3175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:07:52.059298 (+ 4938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:52.059311 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:52.059334 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:52.060176 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:52.060749 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:52.060757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:52.060763 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:52.061072 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:52.061924 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:07:52.061936 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:07:52.062462 (+ 526us) client_negotiation.cc:769] Sending connection context
0811 03:07:52.064437 (+ 1975us) client_negotiation.cc:240] Negotiation successful
0811 03:07:52.065124 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2094,"spinlock_wait_cycles":18304,"thread_start_us":2003,"threads_started":1}
I20260811 03:07:52.069643 9894 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41493
I20260811 03:07:52.070029 9894 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:52.071072 9894 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
I20260811 03:07:52.074056 9654 ts_manager.cc:194] Registered new tserver with Master: e35523f49d5745acbaaae245e804d606 (127.27.109.1:33467)
I20260811 03:07:52.075829 9654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:39513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:07:52.116686 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:50715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41493
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:07:52 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417671, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
W20260811 03:07:52.587180 9903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:07:52.587612 9903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:07:52.587805 9903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:07:52.596230 9903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:07:52.596459 9903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:07:52.596673 9903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:07:52.612901 9903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.109.20:50715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:07:52.615271 9903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:07:52.617120 9903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:07:52.631930 9918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:52.632023 9921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:52.631999 9919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:07:52.633062 9903 server_base.cc:1034] running on GCE node
I20260811 03:07:52.633733 9903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:07:52.634807 9903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:07:52.636037 9903 hybrid_clock.cc:648] HybridClock initialized: now 1786417672635974 us; error 51 us; skew 500 ppm
I20260811 03:07:52.636384 9903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:07:52 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417672, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:52.639943 9903 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:07:52.641776 9903 webserver.cc:460] Webserver started at http://127.27.109.2:45691/ using document root <none> and password file <none>
I20260811 03:07:52.642639 9903 fs_manager.cc:359] Metadata directory not provided
I20260811 03:07:52.642867 9903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:07:52.643267 9903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 03:07:52 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417672, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:52.661774 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "d354c06772e54478a81f1d50be1cd96f"
format_stamp: "Formatted at 2026-08-11 03:07:52 on dist-test-slave-8tdl"
server_key: "519970338b517b6345f1a9f2d9b183c5"
server_key_iv: "75b7fd09206e11e4b28ab6cfa65455c8"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:52.662883 9903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "d354c06772e54478a81f1d50be1cd96f"
format_stamp: "Formatted at 2026-08-11 03:07:52 on dist-test-slave-8tdl"
server_key: "519970338b517b6345f1a9f2d9b183c5"
server_key_iv: "75b7fd09206e11e4b28ab6cfa65455c8"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:52.669181 9903 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:07:52.687220 9928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:52.689352 9903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:07:52.689579 9903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "d354c06772e54478a81f1d50be1cd96f"
format_stamp: "Formatted at 2026-08-11 03:07:52 on dist-test-slave-8tdl"
server_key: "519970338b517b6345f1a9f2d9b183c5"
server_key_iv: "75b7fd09206e11e4b28ab6cfa65455c8"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:52.689891 9903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:07:52.716100 9903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:07:52.720978 9903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:07:52.721331 9903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:07:52.722525 9903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:07:52.724138 9903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:07:52.724224 9903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:52.724390 9903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:07:52.724440 9903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:52.763033 9903 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:39237
I20260811 03:07:52.763055 10049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:39237 every 8 connection(s)
I20260811 03:07:52.764819 9903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:07:52.769358 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 9903
I20260811 03:07:52.769584 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:07:52.770171 28084 external_mini_cluster.cc:1428] Setting key 7bb35a19a17b51496fdb83d8f39ba9ef
Aug 11 03:07:52 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417672, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:07:52.788422 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:52.767835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:60963 (local address 127.27.109.62:41493)
0811 03:07:52.768366 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:52.768372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:52.770771 (+ 2399us) server_negotiation.cc:407] Connection header received
0811 03:07:52.771859 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:52.771864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:52.771982 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:52.772116 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:52.773709 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:52.774510 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:52.776283 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:52.776734 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:52.776894 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:52.780747 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:52.780753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:52.780756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:52.780795 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:52.784292 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:52.785288 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:52.785293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:52.785296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:52.785439 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:52.785974 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:52.785980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:52.785983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:52.786356 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:52.786527 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:52.787700 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0811 03:07:52.788013 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":256,"threads_started":1}
I20260811 03:07:52.790000 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:52.768444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41493 (local address 127.27.109.2:60963)
0811 03:07:52.769378 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:52.769428 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:52.771559 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:52.772389 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:52.772401 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:52.772878 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:07:52.773505 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:52.773523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:52.774727 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:52.774734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:52.776094 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:52.776120 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:52.777010 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:52.777017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:52.777290 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:52.778059 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:07:52.778092 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:07:52.780441 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:07:52.784522 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:52.784534 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:52.784556 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:52.785113 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:52.785571 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:52.785576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:52.785580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:52.785828 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:52.786504 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:07:52.786513 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:07:52.787091 (+ 578us) client_negotiation.cc:769] Sending connection context
0811 03:07:52.787486 (+ 395us) client_negotiation.cc:240] Negotiation successful
0811 03:07:52.788194 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":428,"threads_started":1}
I20260811 03:07:52.792001 10050 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41493
I20260811 03:07:52.792436 10050 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:52.793478 10050 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
I20260811 03:07:52.795668 9654 ts_manager.cc:194] Registered new tserver with Master: d354c06772e54478a81f1d50be1cd96f (127.27.109.2:39237)
I20260811 03:07:52.796944 9654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:60963
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:07:52.847538 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:50715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41493
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:07:53.079493 9894 heartbeater.cc:503] Master 127.27.109.62:41493 was elected leader, sending a full tablet report...
W20260811 03:07:53.228122 10056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:07:53.228497 10056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:07:53.228673 10056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:07:53.236867 10056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:07:53.237033 10056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:07:53.237231 10056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:07:53.248405 10056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.109.20:50715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:07:53.249842 10056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:07:53.251852 10056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:07:53.270411 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:53.270437 10064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:53.270607 10066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:07:53.271450 10056 server_base.cc:1034] running on GCE node
I20260811 03:07:53.272092 10056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:07:53.273176 10056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:07:53.274437 10056 hybrid_clock.cc:648] HybridClock initialized: now 1786417673274350 us; error 63 us; skew 500 ppm
I20260811 03:07:53.275025 10056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:07:53 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417673, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:53.280170 10056 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:07:53.282589 10056 webserver.cc:460] Webserver started at http://127.27.109.3:43899/ using document root <none> and password file <none>
I20260811 03:07:53.283876 10056 fs_manager.cc:359] Metadata directory not provided
I20260811 03:07:53.284041 10056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:07:53.284493 10056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 03:07:53 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417673, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:53.306643 10056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "3bd910be79b043c3bf94dc295c8da10a"
format_stamp: "Formatted at 2026-08-11 03:07:53 on dist-test-slave-8tdl"
server_key: "a1a14535e306c5fb403a63a6d8e8a85d"
server_key_iv: "5ecf2b0690418f70adf531d1dcff9c13"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:53.308090 10056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "3bd910be79b043c3bf94dc295c8da10a"
format_stamp: "Formatted at 2026-08-11 03:07:53 on dist-test-slave-8tdl"
server_key: "a1a14535e306c5fb403a63a6d8e8a85d"
server_key_iv: "5ecf2b0690418f70adf531d1dcff9c13"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:53.315547 10056 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260811 03:07:53.335731 10073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:53.337852 10056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:07:53.338068 10056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "3bd910be79b043c3bf94dc295c8da10a"
format_stamp: "Formatted at 2026-08-11 03:07:53 on dist-test-slave-8tdl"
server_key: "a1a14535e306c5fb403a63a6d8e8a85d"
server_key_iv: "5ecf2b0690418f70adf531d1dcff9c13"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:53.338394 10056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:07:53.364596 10056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:07:53.369690 10056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:07:53.370076 10056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:07:53.371461 10056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:07:53.373422 10056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:07:53.373512 10056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:53.373648 10056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:07:53.373740 10056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:53.423014 10056 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38465
I20260811 03:07:53.423034 10194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38465 every 8 connection(s)
I20260811 03:07:53.424554 10056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:07:53.432933 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 10056
I20260811 03:07:53.433137 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:07:53.433751 28084 external_mini_cluster.cc:1428] Setting key 8b8b6f1fc92cefd16a10498cf2c28277
Aug 11 03:07:53 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417673, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:07:53.447728 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:53.427832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:41131 (local address 127.27.109.62:41493)
0811 03:07:53.428488 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:53.428493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:53.429450 (+ 957us) server_negotiation.cc:407] Connection header received
0811 03:07:53.431568 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:53.431574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:53.431680 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:53.431807 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:53.433548 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.434485 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.436480 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.436980 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.437133 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:53.440864 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:53.440869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:53.440873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:53.440907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:53.444400 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:53.445247 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:53.445251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:53.445254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:53.445419 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:53.445910 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:53.445914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:53.445916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:53.446255 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:53.446431 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:53.447159 (+ 728us) server_negotiation.cc:299] Negotiation successful
0811 03:07:53.447376 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":381,"threads_started":1}
I20260811 03:07:53.449322 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:53.428454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41493 (local address 127.27.109.3:41131)
0811 03:07:53.429279 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:53.429320 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:53.431279 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:53.432088 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:53.432097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:53.432530 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:07:53.433364 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:53.433386 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.434670 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.434715 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:53.436317 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:53.436339 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.437228 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.437237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:53.437495 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:53.438214 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:07:53.438238 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:07:53.440641 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:07:53.444586 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:53.444592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:53.444605 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:53.445128 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:53.445521 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:53.445524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:53.445551 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:53.445786 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:53.446395 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:07:53.446403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:07:53.446807 (+ 404us) client_negotiation.cc:769] Sending connection context
0811 03:07:53.447038 (+ 231us) client_negotiation.cc:240] Negotiation successful
0811 03:07:53.447535 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":359,"threads_started":1}
I20260811 03:07:53.450951 10195 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41493
I20260811 03:07:53.451367 10195 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:53.452421 10195 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
I20260811 03:07:53.454284 9654 ts_manager.cc:194] Registered new tserver with Master: 3bd910be79b043c3bf94dc295c8da10a (127.27.109.3:38465)
I20260811 03:07:53.455214 9654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:41131
I20260811 03:07:53.462626 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:07:53.464584 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 9746
Aug 11 03:07:53 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:07:53.488346 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:53.465773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.27.109.1:33467)
0811 03:07:53.466879 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:53.466885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:53.466912 (+ 27us) server_negotiation.cc:407] Connection header received
0811 03:07:53.466995 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:53.467001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:53.467215 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:53.467386 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:53.469251 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.470843 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.473238 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.473959 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.474078 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:53.480173 (+ 6095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:53.480188 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:53.480204 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:53.480245 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:53.484515 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:53.485466 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:53.485474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:53.485476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:53.485753 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:53.486218 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:53.486223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:53.486227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:53.486664 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:53.487133 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:53.487470 (+ 337us) server_negotiation.cc:299] Negotiation successful
0811 03:07:53.487826 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":713,"threads_started":1}
I20260811 03:07:53.490000 9889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:53.577317 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 9746
I20260811 03:07:53.598904 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 9903
Aug 11 03:07:53 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:07:53.617304 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:53.599819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.27.109.2:39237)
0811 03:07:53.600369 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:53.600377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:53.600404 (+ 27us) server_negotiation.cc:407] Connection header received
0811 03:07:53.600467 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:53.600474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:53.600698 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:53.600831 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:53.602894 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.603928 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.605653 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.606262 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.606372 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:53.611170 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:53.611182 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:53.611199 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:53.611236 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:53.614281 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:53.615130 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:53.615140 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:53.615143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:53.615313 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:53.615811 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:53.615816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:53.615818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:53.616212 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:53.616525 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:53.616750 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 03:07:53.616974 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":285,"threads_started":1}
I20260811 03:07:53.618607 10044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:53.704663 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 9903
I20260811 03:07:53.731976 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 10056
Aug 11 03:07:53 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:07:53.751042 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:53.733154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.27.109.3:38465)
0811 03:07:53.733777 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:53.733784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:53.733814 (+ 30us) server_negotiation.cc:407] Connection header received
0811 03:07:53.733878 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:53.733884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:53.734154 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:53.734287 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:53.735831 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.736874 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.738242 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:53.739050 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:53.739170 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:53.744491 (+ 5321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:53.744506 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:53.744522 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:53.744570 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:53.748104 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:53.748906 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:53.748913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:53.748915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:53.749053 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:53.749435 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:53.749438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:53.749440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:53.749891 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:53.750154 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:53.750383 (+ 229us) server_negotiation.cc:299] Negotiation successful
0811 03:07:53.750613 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":319,"threads_started":1}
I20260811 03:07:53.752526 10190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:53.837997 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 10056
I20260811 03:07:53.858654 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 9618
I20260811 03:07:53.860545 9685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:53.978089 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 9618
I20260811 03:07:54.005818 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41493
--webserver_interface=127.27.109.62
--webserver_port=43431
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41493
--ranger_config_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:50715/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:50715/kms with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:07:54.373468 10208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:07:54.373919 10208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:07:54.374117 10208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:07:54.381942 10208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:07:54.382077 10208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:07:54.382125 10208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:07:54.382180 10208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:07:54.382238 10208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:07:54.393044 10208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:50715/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41493
--ranger_config_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=43431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:07:54.394378 10208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:07:54.396257 10208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:07:54.409929 10216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:54.409969 10215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:54.409999 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:07:54.410988 10208 server_base.cc:1034] running on GCE node
I20260811 03:07:54.411827 10208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:07:54.413286 10208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:07:54.414491 10208 hybrid_clock.cc:648] HybridClock initialized: now 1786417674414431 us; error 45 us; skew 500 ppm
I20260811 03:07:54.414891 10208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:07:54 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417674, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:54.418176 10208 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:07:54.419827 10208 webserver.cc:460] Webserver started at http://127.27.109.62:43431/ using document root <none> and password file <none>
I20260811 03:07:54.420552 10208 fs_manager.cc:359] Metadata directory not provided
I20260811 03:07:54.420713 10208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 03:07:54 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417674, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:54.476562 10208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 03:07:54.480163 10227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:54.481729 10208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 03:07:54.481953 10208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "515de020538e4359942e2092705e3a8f"
format_stamp: "Formatted at 2026-08-11 03:07:50 on dist-test-slave-8tdl"
server_key: "fe4ef223e351f4b737edec3d6d473ed1"
server_key_iv: "bed71e9f1797723d462a883fbfd46d8d"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:54.483099 10208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:07:54.504104 10208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:07:54.508615 10208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:07:54.508949 10208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:07:54.529352 10208 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41493
I20260811 03:07:54.529371 10283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41493 every 8 connection(s)
I20260811 03:07:54.531038 10208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:07:54.538606 10284 sys_catalog.cc:263] Verifying existing consensus state
I20260811 03:07:54.538877 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 10208
I20260811 03:07:54.539815 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:50715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:33467
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:41493
--webserver_port=40583
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:50715/kms with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:07:54.542487 10284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Bootstrap starting.
I20260811 03:07:54.566797 10284 log.cc:824] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Log is configured to *not* fsync() on all Append() calls
I20260811 03:07:54.579924 10284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 03:07:54.580610 10284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Bootstrap complete.
I20260811 03:07:54.594779 10284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } }
I20260811 03:07:54.596033 10284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515de020538e4359942e2092705e3a8f, State: Initialized, Role: FOLLOWER
I20260811 03:07:54.596812 10284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } }
I20260811 03:07:54.597116 10284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:07:54.597254 10284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:07:54.597438 10284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 1 FOLLOWER]: Advancing to term 2
I20260811 03:07:54.603216 10284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } }
I20260811 03:07:54.603843 10284 leader_election.cc:302] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515de020538e4359942e2092705e3a8f; no voters:
I20260811 03:07:54.604530 10284 leader_election.cc:288] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 03:07:54.604696 10289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 2 FOLLOWER]: Leader election won for term 2
I20260811 03:07:54.606112 10289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [term 2 LEADER]: Becoming Leader. State: Replica: 515de020538e4359942e2092705e3a8f, State: Running, Role: LEADER
I20260811 03:07:54.606873 10284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:07:54.608064 10289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } }
I20260811 03:07:54.611159 10290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "515de020538e4359942e2092705e3a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } } }
I20260811 03:07:54.611493 10290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: This master's current role is: LEADER
I20260811 03:07:54.611759 10291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515de020538e4359942e2092705e3a8f. Latest consensus state: current_term: 2 leader_uuid: "515de020538e4359942e2092705e3a8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515de020538e4359942e2092705e3a8f" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41493 } } }
I20260811 03:07:54.612136 10291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f [sys.catalog]: This master's current role is: LEADER
I20260811 03:07:54.621596 10297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:07:54.630527 10297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:07:54.632458 10297 catalog_manager.cc:1259] Loaded cluster ID: 988a7e1f273543159d8e2b217a3f788a
I20260811 03:07:54.632596 10297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:07:54.637885 10297 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:07:54.640005 10297 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Loaded TSK: 0
I20260811 03:07:54.642011 10297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:07:54.648160 10284 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260811 03:07:55.060642 10286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:07:55.061086 10286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:07:55.061256 10286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:07:55.073588 10286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:07:55.073772 10286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:07:55.073959 10286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:07:55.090871 10286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:50715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:33467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=40583
--tserver_master_addrs=127.27.109.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:07:55.092849 10286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:07:55.094776 10286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:07:55.136541 10325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:55.137866 10326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:55.146971 10329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:07:55.147570 10286 server_base.cc:1034] running on GCE node
I20260811 03:07:55.148609 10286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:07:55.155057 10286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:07:55.156458 10286 hybrid_clock.cc:648] HybridClock initialized: now 1786417675156344 us; error 111 us; skew 500 ppm
I20260811 03:07:55.156893 10286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:07:55 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:55.165076 10286 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:07:55.167634 10286 webserver.cc:460] Webserver started at http://127.27.109.1:40583/ using document root <none> and password file <none>
I20260811 03:07:55.168715 10286 fs_manager.cc:359] Metadata directory not provided
I20260811 03:07:55.168879 10286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 03:07:55 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:55.244740 10286 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.008s sys 0.006s
I20260811 03:07:55.248876 10338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:55.250808 10286 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:07:55.251117 10286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "e35523f49d5745acbaaae245e804d606"
format_stamp: "Formatted at 2026-08-11 03:07:51 on dist-test-slave-8tdl"
server_key: "c3bd091e514b3425956d649b4313e10d"
server_key_iv: "2dfaca46aeb019aa90c46bc509cf4dae"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:55.252362 10286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:07:55.273867 10286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:07:55.279163 10286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:07:55.279518 10286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:07:55.280858 10286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:07:55.282809 10286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:07:55.282964 10286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:55.283113 10286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:07:55.283197 10286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:55.338372 10286 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:33467
I20260811 03:07:55.338465 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:33467 every 8 connection(s)
I20260811 03:07:55.340472 10286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:07:55.345916 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 10286
I20260811 03:07:55.346846 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:50715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:39237
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:41493
--webserver_port=45691
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:50715/kms with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:07:55 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:07:55.393298 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:55.344025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:52371 (local address 127.27.109.62:41493)
0811 03:07:55.362324 (+ 18299us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:55.362348 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:55.362416 (+ 68us) server_negotiation.cc:407] Connection header received
0811 03:07:55.362935 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:55.362959 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:55.363518 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:55.364008 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:55.366635 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:55.369069 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:55.372101 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:55.373354 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:55.373585 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:55.379203 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:55.379216 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:55.379240 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:55.379306 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:55.384055 (+ 4749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:55.385380 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:55.385391 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:55.385399 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:55.385657 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:55.386348 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:55.386354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:55.386358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:55.387199 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:55.387417 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:55.391116 (+ 3699us) server_negotiation.cc:299] Negotiation successful
0811 03:07:55.391647 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17918,"thread_start_us":19822,"threads_started":1}
I20260811 03:07:55.393931 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:55.344007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41493 (local address 127.27.109.1:52371)
0811 03:07:55.356608 (+ 12601us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:55.356681 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:55.359143 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:55.364278 (+ 5135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:55.364291 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:55.365257 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:07:55.366279 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:55.366315 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:55.369388 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:55.369395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:55.371810 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:55.371834 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:55.374093 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:55.374102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:55.374408 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:55.375372 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:07:55.375412 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:07:55.378777 (+ 3365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:07:55.384319 (+ 5542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:55.384330 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:55.384352 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:55.385150 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:55.385840 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:55.385845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:55.385850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:55.386177 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:55.387540 (+ 1363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:07:55.387549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:07:55.388044 (+ 495us) client_negotiation.cc:769] Sending connection context
0811 03:07:55.388412 (+ 368us) client_negotiation.cc:240] Negotiation successful
0811 03:07:55.389164 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12248,"thread_start_us":11981,"threads_started":1}
I20260811 03:07:55.399317 10461 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41493
I20260811 03:07:55.399727 10461 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:55.400833 10461 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
W20260811 03:07:55.405242 10461 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:41493 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 03:07:55.405710 10461 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:55.406183 10461 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
I20260811 03:07:55.407932 10461 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:55.408421 10461 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
W20260811 03:07:55.409835 10461 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 03:07:55.832309 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:07:55.832720 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:07:55.832885 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:07:55.844969 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:07:55.845134 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:07:55.845337 10465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:07:55.859223 10465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:50715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:39237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=45691
--tserver_master_addrs=127.27.109.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:07:55.860772 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:07:55.862154 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:07:55.880354 10481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:55.885102 10482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:55.885761 10484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:07:55.886708 10465 server_base.cc:1034] running on GCE node
I20260811 03:07:55.887498 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:07:55.888873 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:07:55.890118 10465 hybrid_clock.cc:648] HybridClock initialized: now 1786417675890039 us; error 62 us; skew 500 ppm
I20260811 03:07:55.890479 10465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:07:55 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:55.894164 10465 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:07:55.900995 10465 webserver.cc:460] Webserver started at http://127.27.109.2:45691/ using document root <none> and password file <none>
I20260811 03:07:55.901973 10465 fs_manager.cc:359] Metadata directory not provided
I20260811 03:07:55.902171 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 03:07:55 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
Aug 11 03:07:55 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:55.945289 10465 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.011s sys 0.007s
I20260811 03:07:55.959508 10492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:55.964434 10465 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.005s sys 0.007s
I20260811 03:07:55.964685 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "d354c06772e54478a81f1d50be1cd96f"
format_stamp: "Formatted at 2026-08-11 03:07:52 on dist-test-slave-8tdl"
server_key: "519970338b517b6345f1a9f2d9b183c5"
server_key_iv: "75b7fd09206e11e4b28ab6cfa65455c8"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:55.966264 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:07:56.022667 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:07:56.027877 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:07:56.028224 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:07:56.029376 10465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:07:56.031059 10465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:07:56.031193 10465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:56.031347 10465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:07:56.031443 10465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:56.083850 10465 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:39237
I20260811 03:07:56.083899 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:39237 every 8 connection(s)
I20260811 03:07:56.085467 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:07:56.087518 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 10465
I20260811 03:07:56.088408 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.109.20:50715/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:38465
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:41493
--webserver_port=43899
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:50715/kms with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:07:56 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:07:56.128101 10620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:56.089971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41493 (local address 127.27.109.2:59969)
0811 03:07:56.098886 (+ 8915us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:56.098933 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:56.100724 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:56.103147 (+ 2423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:56.103165 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:56.103984 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:07:56.105014 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:56.105047 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.106855 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.106864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:56.109606 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:56.109644 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.111324 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.111335 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:56.111744 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:56.112731 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:07:56.112771 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:07:56.116106 (+ 3335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:07:56.121619 (+ 5513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:56.121634 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:56.121660 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:56.122534 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:56.123166 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:56.123174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:56.123180 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:56.123534 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:56.124506 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:07:56.124518 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:07:56.125102 (+ 584us) client_negotiation.cc:769] Sending connection context
0811 03:07:56.125510 (+ 408us) client_negotiation.cc:240] Negotiation successful
0811 03:07:56.126258 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8535,"thread_start_us":11018,"threads_started":1}
I20260811 03:07:56.129544 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:56.094599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:59969 (local address 127.27.109.62:41493)
0811 03:07:56.102386 (+ 7787us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:56.102393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:56.102417 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:07:56.102492 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:56.102497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:56.102590 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:56.102773 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:56.105289 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.106546 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.109848 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.110765 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.110933 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:56.116541 (+ 5608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:56.116552 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:56.116561 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:56.116619 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:56.121332 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:56.122766 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:56.122776 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:56.122782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:56.122983 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:56.123713 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:56.123723 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:56.123729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:56.124203 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:56.124384 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:56.128754 (+ 4370us) server_negotiation.cc:299] Negotiation successful
0811 03:07:56.129087 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7634,"thread_start_us":7499,"threads_started":1}
I20260811 03:07:56.131908 10618 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41493
I20260811 03:07:56.132198 10618 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:56.133157 10618 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
I20260811 03:07:56.136015 10244 ts_manager.cc:194] Registered new tserver with Master: d354c06772e54478a81f1d50be1cd96f (127.27.109.2:39237)
I20260811 03:07:56.138576 10244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:59969
I20260811 03:07:56.410013 10461 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:56.410516 10461 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
I20260811 03:07:56.411859 10244 ts_manager.cc:194] Registered new tserver with Master: e35523f49d5745acbaaae245e804d606 (127.27.109.1:33467)
I20260811 03:07:56.413322 10244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:52371
Aug 11 03:07:56 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
W20260811 03:07:56.599339 10622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:07:56.599711 10622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:07:56.599880 10622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:07:56.607604 10622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:07:56.607767 10622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:07:56.607985 10622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:07:56.618427 10622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.109.20:50715/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:38465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=43899
--tserver_master_addrs=127.27.109.62:41493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:07:56.619872 10622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:07:56.621682 10622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:07:56.642395 10630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:56.643572 10629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:07:56.656589 10632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:07:56.657011 10622 server_base.cc:1034] running on GCE node
I20260811 03:07:56.658066 10622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:07:56.659742 10622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:07:56.661110 10622 hybrid_clock.cc:648] HybridClock initialized: now 1786417676661034 us; error 70 us; skew 500 ppm
I20260811 03:07:56.661517 10622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:07:56 dist-test-slave-8tdl krb5kdc[9141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:07:56.666302 10622 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:07:56.668686 10622 webserver.cc:460] Webserver started at http://127.27.109.3:43899/ using document root <none> and password file <none>
I20260811 03:07:56.669770 10622 fs_manager.cc:359] Metadata directory not provided
I20260811 03:07:56.669970 10622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 03:07:56 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:56.704234 10622 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.005s sys 0.005s
I20260811 03:07:56.708546 10640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:56.710574 10622 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260811 03:07:56.710794 10622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "3bd910be79b043c3bf94dc295c8da10a"
format_stamp: "Formatted at 2026-08-11 03:07:53 on dist-test-slave-8tdl"
server_key: "a1a14535e306c5fb403a63a6d8e8a85d"
server_key_iv: "5ecf2b0690418f70adf531d1dcff9c13"
server_key_version: "kuduclusterkey@0"
I20260811 03:07:56.712308 10622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:07:56.757182 10622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:07:56.763741 10622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:07:56.764218 10622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:07:56.768460 10622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:07:56.770996 10622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:07:56.771157 10622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:07:56.771309 10622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:07:56.771502 10622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 11 03:07:56 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417675, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for HTTP/127.27.109.20@KRBTEST.COM
I20260811 03:07:56.917330 10622 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38465
I20260811 03:07:56.917403 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38465 every 8 connection(s)
I20260811 03:07:56.919538 10622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:07:56.925107 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 10622
Aug 11 03:07:56 dist-test-slave-8tdl krb5kdc[9141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417676, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:07:56.983521 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:56.924469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:53237 (local address 127.27.109.62:41493)
0811 03:07:56.954506 (+ 30037us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:56.954513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:56.958429 (+ 3916us) server_negotiation.cc:407] Connection header received
0811 03:07:56.958520 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:56.958526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:56.958649 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:56.958800 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:56.963635 (+ 4835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.964925 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.969766 (+ 4841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.970491 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.970601 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:56.975551 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:56.975558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:56.975564 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:56.975624 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:56.979308 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:56.980390 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:56.980396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:56.980400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:56.980550 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:56.981136 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:56.981141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:56.981145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:56.981518 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:56.981649 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:56.982775 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0811 03:07:56.983044 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29903,"thread_start_us":29766,"threads_started":1}
I20260811 03:07:56.986239 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:56.923361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41493 (local address 127.27.109.3:53237)
0811 03:07:56.955235 (+ 31874us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:56.955316 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:56.957958 (+ 2642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:56.959325 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:56.959340 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:56.960093 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:07:56.963328 (+ 3235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:56.963386 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.967268 (+ 3882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.967274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:56.969548 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:56.969570 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.971020 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.971027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:56.971328 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:56.972221 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:07:56.972255 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:07:56.975262 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:07:56.979536 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:56.979547 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:56.979568 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:56.980229 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:56.980698 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:07:56.980703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:07:56.980707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:07:56.980984 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:07:56.981756 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:07:56.981765 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:07:56.982230 (+ 465us) client_negotiation.cc:769] Sending connection context
0811 03:07:56.984090 (+ 1860us) client_negotiation.cc:240] Negotiation successful
0811 03:07:56.984731 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31494,"thread_start_us":31268,"threads_started":1}
I20260811 03:07:56.988642 10769 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41493
I20260811 03:07:56.989008 10769 heartbeater.cc:464] Registering TS with master...
I20260811 03:07:56.990095 10769 heartbeater.cc:511] Master 127.27.109.62:41493 requested a full tablet report, sending...
I20260811 03:07:56.992120 10244 ts_manager.cc:194] Registered new tserver with Master: 3bd910be79b043c3bf94dc295c8da10a (127.27.109.3:38465)
I20260811 03:07:56.993279 10244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:53237
I20260811 03:07:56.995844 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:56.954478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.27.109.62:41493)
0811 03:07:56.959812 (+ 5334us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:56.959818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:56.959839 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:07:56.959888 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:56.959896 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:56.960009 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:56.960147 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:56.962377 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.963956 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.972078 (+ 8122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:56.973665 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:56.973832 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:56.981535 (+ 7703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:56.981543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:56.981548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:56.981602 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:56.988843 (+ 7241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:56.990008 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:56.990013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:56.990016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:56.990204 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:56.991660 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:56.991666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:56.991669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:56.992123 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:56.992427 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:56.995158 (+ 2731us) server_negotiation.cc:299] Negotiation successful
0811 03:07:56.995451 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5199,"thread_start_us":5831,"threads_started":1}
I20260811 03:07:56.999693 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:07:57.055698 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.021439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.27.109.62:41493)
0811 03:07:57.021741 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.021749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.035399 (+ 13650us) server_negotiation.cc:407] Connection header received
0811 03:07:57.035685 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.035693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.035805 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.035915 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:57.038426 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.039660 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.041996 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.042845 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.042963 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.047173 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:57.047180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:57.047186 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:57.047237 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:57.051629 (+ 4392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:57.052659 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:57.052667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:57.052672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:57.052863 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:57.053387 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:57.053393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:57.053397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:57.053978 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:57.054250 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.055018 (+ 768us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.055296 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260811 03:07:57.063858 10244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:07:57.077270 10244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:07:57.143985 10618 heartbeater.cc:503] Master 127.27.109.62:41493 was elected leader, sending a full tablet report...
I20260811 03:07:57.146270 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.109612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35360)
0811 03:07:57.135012 (+ 25400us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:57.135034 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:57.135175 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:57.136552 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:57.136560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:57.136591 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:07:57.137044 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.137059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.139464 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.139470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.142113 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.142132 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.145226 (+ 3094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.145233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.145467 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.145525 (+ 58us) client_negotiation.cc:769] Sending connection context
0811 03:07:57.145655 (+ 130us) client_negotiation.cc:240] Negotiation successful
0811 03:07:57.145882 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25259,"thread_start_us":27910,"threads_started":1}
I20260811 03:07:57.147248 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.110350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.27.109.1:33467)
0811 03:07:57.134946 (+ 24596us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.134956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.135325 (+ 369us) server_negotiation.cc:407] Connection header received
0811 03:07:57.135465 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.135474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.135792 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.136028 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:07:57.137270 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.139148 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.142333 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.143942 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.144066 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.144992 (+ 926us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.146657 (+ 1665us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.146907 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24304,"thread_start_us":24616,"threads_started":1}
I20260811 03:07:57.150121 10789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.131330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39237 (local address 127.0.0.1:36908)
0811 03:07:57.135639 (+ 4309us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:57.135656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:57.135763 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:57.138875 (+ 3112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:57.138881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:57.138910 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:07:57.139292 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.139308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.144211 (+ 4903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.144216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.146368 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.146387 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.149018 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.149027 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.149248 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.149288 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 03:07:57.149398 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 03:07:57.149566 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4195,"mutex_wait_us":1,"thread_start_us":4111,"threads_started":1}
I20260811 03:07:57.151145 10791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.123991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.27.109.2:39237)
0811 03:07:57.137853 (+ 13862us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.137861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.137950 (+ 89us) server_negotiation.cc:407] Connection header received
0811 03:07:57.138039 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.138047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.138252 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.138568 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:07:57.142430 (+ 3862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.144039 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.146593 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.148123 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.148270 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.148519 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.150537 (+ 2018us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.150771 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13225,"thread_start_us":13072,"threads_started":1}
I20260811 03:07:57.153090 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.109340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38465 (local address 127.0.0.1:45396)
0811 03:07:57.121988 (+ 12648us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:57.122054 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:57.122287 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:57.139540 (+ 17253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:57.139545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:57.139584 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:07:57.140012 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.140029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.142350 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.142357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.144679 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.144697 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.152308 (+ 7611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.152316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.152513 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.152547 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:07:57.152623 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 03:07:57.152771 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12266,"thread_start_us":12083,"threads_started":1}
I20260811 03:07:57.153978 10389 tablet_service.cc:1467] Processing CreateTablet for tablet 0d26d9df785b4526a1764a3b5d4b8e82 (DEFAULT_TABLE table=test-table [id=bab04c9095a248d5909c421ae1124e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:07:57.155843 10389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d26d9df785b4526a1764a3b5d4b8e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:07:57.156499 10548 tablet_service.cc:1467] Processing CreateTablet for tablet 0d26d9df785b4526a1764a3b5d4b8e82 (DEFAULT_TABLE table=test-table [id=bab04c9095a248d5909c421ae1124e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:07:57.159485 10548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d26d9df785b4526a1764a3b5d4b8e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:07:57.171278 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.109519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.27.109.3:38465)
0811 03:07:57.136367 (+ 26848us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.136377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.136409 (+ 32us) server_negotiation.cc:407] Connection header received
0811 03:07:57.136502 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.136509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.136759 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.140226 (+ 3467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:07:57.140613 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.142191 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.148644 (+ 6453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.152040 (+ 3396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.169970 (+ 17930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.170361 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.170599 (+ 238us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.170849 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26647,"thread_start_us":26727,"threads_started":1}
I20260811 03:07:57.175683 10696 tablet_service.cc:1467] Processing CreateTablet for tablet 0d26d9df785b4526a1764a3b5d4b8e82 (DEFAULT_TABLE table=test-table [id=bab04c9095a248d5909c421ae1124e46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:07:57.177563 10696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d26d9df785b4526a1764a3b5d4b8e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:07:57.177793 10795 tablet_bootstrap.cc:492] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: Bootstrap starting.
I20260811 03:07:57.185808 10795 tablet_bootstrap.cc:654] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: Neither blocks nor log segments found. Creating new log.
I20260811 03:07:57.188560 10795 log.cc:824] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: Log is configured to *not* fsync() on all Append() calls
I20260811 03:07:57.199774 10798 tablet_bootstrap.cc:492] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Bootstrap starting.
I20260811 03:07:57.207140 10804 tablet_bootstrap.cc:492] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Bootstrap starting.
I20260811 03:07:57.208271 10795 tablet_bootstrap.cc:492] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: No bootstrap required, opened a new log
I20260811 03:07:57.208652 10798 tablet_bootstrap.cc:654] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Neither blocks nor log segments found. Creating new log.
I20260811 03:07:57.208963 10795 ts_tablet_manager.cc:1397] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.012s
I20260811 03:07:57.211547 10798 log.cc:824] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Log is configured to *not* fsync() on all Append() calls
I20260811 03:07:57.216089 10804 tablet_bootstrap.cc:654] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Neither blocks nor log segments found. Creating new log.
I20260811 03:07:57.220762 10804 log.cc:824] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Log is configured to *not* fsync() on all Append() calls
I20260811 03:07:57.221355 10798 tablet_bootstrap.cc:492] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: No bootstrap required, opened a new log
I20260811 03:07:57.221887 10798 ts_tablet_manager.cc:1397] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.005s
I20260811 03:07:57.227221 10795 raft_consensus.cc:348] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.228222 10795 raft_consensus.cc:374] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:07:57.228528 10795 raft_consensus.cc:729] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35523f49d5745acbaaae245e804d606, State: Initialized, Role: FOLLOWER
I20260811 03:07:57.229653 10795 consensus_queue.cc:260] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.232002 10804 tablet_bootstrap.cc:492] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: No bootstrap required, opened a new log
I20260811 03:07:57.232519 10804 ts_tablet_manager.cc:1397] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.011s
I20260811 03:07:57.239466 10795 ts_tablet_manager.cc:1428] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: Time spent starting tablet: real 0.030s user 0.026s sys 0.001s
I20260811 03:07:57.240031 10461 heartbeater.cc:503] Master 127.27.109.62:41493 was elected leader, sending a full tablet report...
I20260811 03:07:57.240151 10798 raft_consensus.cc:348] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.240767 10798 raft_consensus.cc:374] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:07:57.240931 10798 raft_consensus.cc:729] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d354c06772e54478a81f1d50be1cd96f, State: Initialized, Role: FOLLOWER
I20260811 03:07:57.241861 10798 consensus_queue.cc:260] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.249295 10804 raft_consensus.cc:348] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.250144 10804 raft_consensus.cc:374] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:07:57.250355 10804 raft_consensus.cc:729] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd910be79b043c3bf94dc295c8da10a, State: Initialized, Role: FOLLOWER
I20260811 03:07:57.251610 10804 consensus_queue.cc:260] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.253863 10769 heartbeater.cc:503] Master 127.27.109.62:41493 was elected leader, sending a full tablet report...
I20260811 03:07:57.255223 10798 ts_tablet_manager.cc:1428] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Time spent starting tablet: real 0.033s user 0.023s sys 0.012s
I20260811 03:07:57.257243 10804 ts_tablet_manager.cc:1428] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Time spent starting tablet: real 0.025s user 0.005s sys 0.014s
W20260811 03:07:57.344588 10619 tablet.cc:2369] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:07:57.349599 10462 tablet.cc:2369] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:07:57.427812 10770 tablet.cc:2369] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:07:57.555365 10809 raft_consensus.cc:482] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:07:57.555919 10809 raft_consensus.cc:504] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.558921 10809 leader_election.cc:288] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bd910be79b043c3bf94dc295c8da10a (127.27.109.3:38465), e35523f49d5745acbaaae245e804d606 (127.27.109.1:33467)
I20260811 03:07:57.571543 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.559707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39663 (local address 127.27.109.1:33467)
0811 03:07:57.559943 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.559949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.561630 (+ 1681us) server_negotiation.cc:407] Connection header received
0811 03:07:57.561916 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.561920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.562025 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.562127 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:07:57.563727 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.564729 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.568093 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.569958 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.570188 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.570306 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.571022 (+ 716us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.571195 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 03:07:57.572201 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.559525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:49681 (local address 127.27.109.3:38465)
0811 03:07:57.559790 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.559801 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.561562 (+ 1761us) server_negotiation.cc:407] Connection header received
0811 03:07:57.561814 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.561820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.561915 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.562039 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:07:57.563603 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.564735 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.568752 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.570631 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.570921 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.571070 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.571612 (+ 542us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.571762 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 03:07:57.572641 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.559127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38465 (local address 127.27.109.2:49681)
0811 03:07:57.561376 (+ 2249us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:57.561400 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:57.561604 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:57.562091 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:57.562100 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:57.562131 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:07:57.563262 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.563280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.564920 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.564925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.568464 (+ 3539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.568485 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.571072 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.571082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.571334 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.571361 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 03:07:57.571580 (+ 219us) client_negotiation.cc:240] Negotiation successful
0811 03:07:57.571750 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2121,"mutex_wait_us":74,"thread_start_us":1304,"threads_started":1}
I20260811 03:07:57.573351 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.559092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.27.109.2:39663)
0811 03:07:57.561372 (+ 2280us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:57.561400 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:07:57.561673 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:07:57.562424 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:07:57.562429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:07:57.562498 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:07:57.563481 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.563497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.564950 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.564956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.567840 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:07:57.567862 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.570415 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.570424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:07:57.570764 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.570808 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 03:07:57.572177 (+ 1369us) client_negotiation.cc:240] Negotiation successful
0811 03:07:57.572351 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2022,"spinlock_wait_cycles":44800,"thread_start_us":1682,"threads_started":1}
I20260811 03:07:57.574340 10718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d26d9df785b4526a1764a3b5d4b8e82" candidate_uuid: "d354c06772e54478a81f1d50be1cd96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd910be79b043c3bf94dc295c8da10a" is_pre_election: true
I20260811 03:07:57.575129 10718 raft_consensus.cc:2454] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d354c06772e54478a81f1d50be1cd96f in term 0.
I20260811 03:07:57.575356 10411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d26d9df785b4526a1764a3b5d4b8e82" candidate_uuid: "d354c06772e54478a81f1d50be1cd96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35523f49d5745acbaaae245e804d606" is_pre_election: true
I20260811 03:07:57.576143 10496 leader_election.cc:302] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd910be79b043c3bf94dc295c8da10a, d354c06772e54478a81f1d50be1cd96f; no voters:
I20260811 03:07:57.576167 10411 raft_consensus.cc:2454] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d354c06772e54478a81f1d50be1cd96f in term 0.
I20260811 03:07:57.576814 10809 raft_consensus.cc:2781] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:07:57.577114 10809 raft_consensus.cc:482] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:07:57.577260 10809 raft_consensus.cc:3037] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:07:57.581372 10809 raft_consensus.cc:504] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.582388 10809 leader_election.cc:288] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [CANDIDATE]: Term 1 election: Requested vote from peers 3bd910be79b043c3bf94dc295c8da10a (127.27.109.3:38465), e35523f49d5745acbaaae245e804d606 (127.27.109.1:33467)
I20260811 03:07:57.583237 10718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d26d9df785b4526a1764a3b5d4b8e82" candidate_uuid: "d354c06772e54478a81f1d50be1cd96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd910be79b043c3bf94dc295c8da10a"
I20260811 03:07:57.583410 10411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d26d9df785b4526a1764a3b5d4b8e82" candidate_uuid: "d354c06772e54478a81f1d50be1cd96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e35523f49d5745acbaaae245e804d606"
I20260811 03:07:57.583645 10718 raft_consensus.cc:3037] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:07:57.583807 10411 raft_consensus.cc:3037] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:07:57.589007 10411 raft_consensus.cc:2454] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d354c06772e54478a81f1d50be1cd96f in term 1.
I20260811 03:07:57.589071 10718 raft_consensus.cc:2454] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d354c06772e54478a81f1d50be1cd96f in term 1.
I20260811 03:07:57.589903 10497 leader_election.cc:302] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bd910be79b043c3bf94dc295c8da10a, d354c06772e54478a81f1d50be1cd96f, e35523f49d5745acbaaae245e804d606; no voters:
I20260811 03:07:57.590492 10809 raft_consensus.cc:2781] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:07:57.591495 10809 raft_consensus.cc:686] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 1 LEADER]: Becoming Leader. State: Replica: d354c06772e54478a81f1d50be1cd96f, State: Running, Role: LEADER
I20260811 03:07:57.592182 10809 consensus_queue.cc:237] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } }
I20260811 03:07:57.598596 10243 catalog_manager.cc:5495] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f reported cstate change: term changed from 0 to 1, leader changed from <none> to d354c06772e54478a81f1d50be1cd96f (127.27.109.2). New cstate: current_term: 1 leader_uuid: "d354c06772e54478a81f1d50be1cd96f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d354c06772e54478a81f1d50be1cd96f" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 } health_report { overall_health: UNKNOWN } } }
I20260811 03:07:57.707691 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.698466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.27.109.2:39237)
0811 03:07:57.699542 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.699549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.699573 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:07:57.699915 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.699920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.700010 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.700252 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:07:57.700977 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.702017 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.704508 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.705273 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.705439 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.706157 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:07:57.706896 (+ 739us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:07:57.707087 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.707147 (+ 60us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.707234 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":496,"threads_started":1}
I20260811 03:07:57.719146 10411 raft_consensus.cc:1261] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 1 FOLLOWER]: Refusing update from remote peer d354c06772e54478a81f1d50be1cd96f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:07:57.719146 10718 raft_consensus.cc:1261] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 1 FOLLOWER]: Refusing update from remote peer d354c06772e54478a81f1d50be1cd96f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:07:57.720824 10813 consensus_queue.cc:1035] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bd910be79b043c3bf94dc295c8da10a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:07:57.721450 10809 consensus_queue.cc:1035] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e35523f49d5745acbaaae245e804d606" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:07:57.750909 10819 mvcc.cc:205] Tried to move back new op lower bound from 7317166807923011584 to 7317166807422459904. Current Snapshot: MvccSnapshot[applied={T|T < 7317166807923011584 or (T in {7317166807923011584})}]
I20260811 03:07:57.751792 10820 mvcc.cc:205] Tried to move back new op lower bound from 7317166807923011584 to 7317166807422459904. Current Snapshot: MvccSnapshot[applied={T|T < 7317166807923011584}]
I20260811 03:07:57.767194 10243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:07:57.767712 10243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:07:57.773316 10243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 515de020538e4359942e2092705e3a8f: Sending DeleteTablet for 3 replicas of tablet 0d26d9df785b4526a1764a3b5d4b8e82
I20260811 03:07:57.775362 10696 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d26d9df785b4526a1764a3b5d4b8e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:07:57 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:45396
I20260811 03:07:57.775431 10548 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d26d9df785b4526a1764a3b5d4b8e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:07:57 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36908
I20260811 03:07:57.775857 10389 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d26d9df785b4526a1764a3b5d4b8e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:07:57 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:35360
I20260811 03:07:57.781360 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 10286
I20260811 03:07:57.787916 10832 tablet_replica.cc:330] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: stopping tablet replica
I20260811 03:07:57.788843 10832 raft_consensus.cc:2229] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:07:57.796123 10831 tablet_replica.cc:330] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: stopping tablet replica
I20260811 03:07:57.796739 10831 raft_consensus.cc:2229] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:07:57.797385 10831 raft_consensus.cc:2258] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:07:57.803380 10832 raft_consensus.cc:2258] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:07:57.804389 10831 ts_tablet_manager.cc:1905] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:07:57.805970 10833 tablet_replica.cc:330] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: stopping tablet replica
I20260811 03:07:57.806635 10833 raft_consensus.cc:2229] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:07:57.807225 10832 ts_tablet_manager.cc:1905] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:07:57.807646 10833 raft_consensus.cc:2258] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:07:57.817085 10831 ts_tablet_manager.cc:1918] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:07:57.817543 10831 log.cc:1196] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/0d26d9df785b4526a1764a3b5d4b8e82
I20260811 03:07:57.817430 10833 ts_tablet_manager.cc:1905] T 0d26d9df785b4526a1764a3b5d4b8e82 P e35523f49d5745acbaaae245e804d606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:07:57.818409 10832 ts_tablet_manager.cc:1918] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:07:57.818461 10831 ts_tablet_manager.cc:1939] T 0d26d9df785b4526a1764a3b5d4b8e82 P 3bd910be79b043c3bf94dc295c8da10a: Deleting consensus metadata
I20260811 03:07:57.818773 10832 log.cc:1196] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/0d26d9df785b4526a1764a3b5d4b8e82
I20260811 03:07:57.819867 10832 ts_tablet_manager.cc:1939] T 0d26d9df785b4526a1764a3b5d4b8e82 P d354c06772e54478a81f1d50be1cd96f: Deleting consensus metadata
I20260811 03:07:57.821471 10230 catalog_manager.cc:4841] TS 3bd910be79b043c3bf94dc295c8da10a (127.27.109.3:38465): tablet 0d26d9df785b4526a1764a3b5d4b8e82 (table test-table [id=bab04c9095a248d5909c421ae1124e46]) successfully deleted
I20260811 03:07:57.822592 10230 catalog_manager.cc:4841] TS d354c06772e54478a81f1d50be1cd96f (127.27.109.2:39237): tablet 0d26d9df785b4526a1764a3b5d4b8e82 (table test-table [id=bab04c9095a248d5909c421ae1124e46]) successfully deleted
I20260811 03:07:57.823170 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.783803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.27.109.1:33467)
0811 03:07:57.784068 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.784075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.794441 (+ 10366us) server_negotiation.cc:407] Connection header received
0811 03:07:57.794808 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.794815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.794974 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.795145 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:57.799904 (+ 4759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.801374 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.803280 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.804678 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.804822 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.809649 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:57.809675 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:57.809694 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:57.809741 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:57.816333 (+ 6592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:57.817978 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:57.817989 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:57.817994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:57.818287 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:57.820087 (+ 1800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:57.820094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:57.820098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:57.820872 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:57.821555 (+ 683us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.821929 (+ 374us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.822236 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 03:07:57.824375 10456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:57.928085 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 10286
I20260811 03:07:57.960163 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 10465
W20260811 03:07:57.960282 10231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:33467 (error 108)
I20260811 03:07:57.963656 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.962207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35388)
0811 03:07:57.963259 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:57.963401 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":663,"threads_started":1}
W20260811 03:07:57.964085 10231 catalog_manager.cc:4584] TS e35523f49d5745acbaaae245e804d606 (127.27.109.1:33467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d26d9df785b4526a1764a3b5d4b8e82: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
I20260811 03:07:57.979934 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:57.961271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.27.109.2:39237)
0811 03:07:57.961542 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:57.961550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:57.961680 (+ 130us) server_negotiation.cc:407] Connection header received
0811 03:07:57.961929 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:57.961936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:57.962039 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:57.962161 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:57.964369 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.965610 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.967644 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:57.968377 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:57.968519 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:57.971719 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:57.971737 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:57.971759 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:57.971833 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:57.976450 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:57.977408 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:57.977418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:57.977422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:57.977597 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:57.978016 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:57.978021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:57.978024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:57.978512 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:57.978864 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:57.979233 (+ 369us) server_negotiation.cc:299] Negotiation successful
0811 03:07:57.979533 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260811 03:07:57.980923 10613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:58.018400 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:58.017681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35404)
0811 03:07:58.017933 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:58.018162 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260811 03:07:58.019819 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:58.019344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35414)
0811 03:07:58.019573 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:58.019656 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260811 03:07:58.080551 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:58.079963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35422)
0811 03:07:58.080220 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:58.080366 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260811 03:07:58.081678 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:58.081281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35436)
0811 03:07:58.081441 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:58.081524 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 03:07:58.103219 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 10465
I20260811 03:07:58.126273 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 10622
I20260811 03:07:58.143376 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:58.127208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.27.109.3:38465)
0811 03:07:58.127794 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0811 03:07:58.127798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:07:58.127815 (+ 17us) server_negotiation.cc:407] Connection header received
0811 03:07:58.127881 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:07:58.127885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:07:58.127958 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:07:58.128058 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:07:58.129722 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:58.130630 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:58.132045 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:07:58.132511 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:07:58.132677 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:07:58.135535 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:07:58.135550 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:07:58.135567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:07:58.135609 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:07:58.139214 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:58.140059 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:58.140074 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:58.140077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:58.140316 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:07:58.140944 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:07:58.140950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:07:58.140953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:07:58.141614 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:07:58.141939 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0811 03:07:58.142478 (+ 539us) server_negotiation.cc:299] Negotiation successful
0811 03:07:58.142940 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":342,"threads_started":1}
I20260811 03:07:58.144435 10764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:58.162547 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:58.162007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35440)
0811 03:07:58.162291 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:58.162382 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260811 03:07:58.163681 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:58.163261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35456)
0811 03:07:58.163455 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:58.163544 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260811 03:07:58.240448 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 10622
I20260811 03:07:58.262861 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 10208
I20260811 03:07:58.264333 10279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:07:58.380647 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:07:58.379836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:33467 (local address 127.0.0.1:35462)
0811 03:07:58.380227 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0811 03:07:58.380338 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:33467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260811 03:07:58.380963 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 10208
I20260811 03:07:58.412674 28084 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 03:07:58.759305 28084 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 03:07:58.759595 28084 mini_ranger.cc:66] Stopping Ranger...
I20260811 03:07:59.116612 28084 mini_ranger.cc:68] Stopped Ranger
2026-08-11 03:07:59.116 UTC [9173] LOG: received smart shutdown request
2026-08-11 03:07:59.120 UTC [9173] LOG: background worker "logical replication launcher" (PID 9182) exited with exit code 1
2026-08-11 03:07:59.120 UTC [9177] LOG: shutting down
2026-08-11 03:07:59.348 UTC [9173] LOG: database system is shut down
2026-08-11T03:07:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58182 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:07:59 dist-test-slave-8tdl krb5kdc[10853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:07:59 dist-test-slave-8tdl krb5kdc[10853](info): set up 2 sockets
Aug 11 03:07:59 dist-test-slave-8tdl krb5kdc[10853](info): commencing operation
krb5kdc: starting...
W20260811 03:08:00.686121 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:08:00 dist-test-slave-8tdl krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:08:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:08:01.003662 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37453
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:37453,127.27.109.61:43943,127.27.109.60:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:01.375874 10866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:01.376281 10866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:01.376444 10866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:01.384027 10866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:01.384210 10866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:01.384325 10866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:01.384433 10866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:01.384526 10866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:01.394992 10866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:37453,127.27.109.61:43943,127.27.109.60:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:01.396481 10866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:01.397949 10866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:01.410931 10872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:01.410878 10873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:01.410902 10876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:01.412322 10866 server_base.cc:1034] running on GCE node
I20260811 03:08:01.413419 10866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:01.415278 10866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:01.416615 10866 hybrid_clock.cc:648] HybridClock initialized: now 1786417681416566 us; error 36 us; skew 500 ppm
I20260811 03:08:01.416987 10866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:01 dist-test-slave-8tdl krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:01.424849 10866 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:08:01.426923 10866 webserver.cc:460] Webserver started at http://127.27.109.62:45959/ using document root <none> and password file <none>
I20260811 03:08:01.427788 10866 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:01.427979 10866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:01.428391 10866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:01.432055 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "1f22684fde4c460eba922a5bb62a5216"
format_stamp: "Formatted at 2026-08-11 03:08:01 on dist-test-slave-8tdl"
server_key: "c4983c3af8ba23ce3055c6d226919dca"
server_key_iv: "902cf7dba4722f853148370052a7fcfd"
server_key_version: "encryptionkey@0"
I20260811 03:08:01.433028 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "1f22684fde4c460eba922a5bb62a5216"
format_stamp: "Formatted at 2026-08-11 03:08:01 on dist-test-slave-8tdl"
server_key: "c4983c3af8ba23ce3055c6d226919dca"
server_key_iv: "902cf7dba4722f853148370052a7fcfd"
server_key_version: "encryptionkey@0"
I20260811 03:08:01.439347 10866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260811 03:08:01.443430 10884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:01.445037 10866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:08:01.445266 10866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "1f22684fde4c460eba922a5bb62a5216"
format_stamp: "Formatted at 2026-08-11 03:08:01 on dist-test-slave-8tdl"
server_key: "c4983c3af8ba23ce3055c6d226919dca"
server_key_iv: "902cf7dba4722f853148370052a7fcfd"
server_key_version: "encryptionkey@0"
I20260811 03:08:01.445529 10866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:01.465261 10866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:01.470233 10866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:01.470577 10866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:01.490614 10866 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:37453
I20260811 03:08:01.490660 10941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:37453 every 8 connection(s)
I20260811 03:08:01.492163 10866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:08:01.496690 10942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:01.497385 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 10866
I20260811 03:08:01.497694 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:08:01.498421 28084 external_mini_cluster.cc:1428] Setting key eeb21610d29009e41a7fecf80cbbb7e0
I20260811 03:08:01.503902 10942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:08:01 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.61@KRBTEST.COM" created.
I20260811 03:08:01.529691 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.507320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.27.109.62:37453)
0811 03:08:01.509349 (+ 2029us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:01.509357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:01.509424 (+ 67us) server_negotiation.cc:407] Connection header received
0811 03:08:01.511178 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:01.511184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:01.511436 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:01.511652 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:01.513036 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:01.514298 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:01.515866 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:01.516462 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:01.516675 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:01.521037 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:01.521047 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:01.521062 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:01.521099 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:01.524389 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:01.525450 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:01.525457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:01.525459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:01.525661 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:01.526166 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:01.526170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:01.526174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:01.526832 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:01.527054 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:01.527642 (+ 588us) server_negotiation.cc:299] Negotiation successful
0811 03:08:01.528051 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1801,"spinlock_wait_cycles":46336,"thread_start_us":1578,"threads_started":1}
I20260811 03:08:01.529703 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.507319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37453 (local address 127.0.0.1:52656)
0811 03:08:01.508283 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.508346 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:01.510772 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:01.511678 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:01.511685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:01.512443 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:01.512807 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:01.512826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:01.514484 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:01.514491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:01.515709 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:01.515722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:01.516721 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:01.516728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:01.516916 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:01.517682 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:01.517706 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:01.520737 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:01.524562 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:01.524579 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:01.524596 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:01.525257 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:01.525788 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:01.525794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:01.525798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:01.526035 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:01.526975 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:01.526983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:01.527236 (+ 253us) client_negotiation.cc:769] Sending connection context
0811 03:08:01.527500 (+ 264us) client_negotiation.cc:240] Negotiation successful
0811 03:08:01.528193 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":167,"thread_start_us":425,"threads_started":1}
I20260811 03:08:01.533010 10942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:08:01.534332 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.533848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46364)
0811 03:08:01.534065 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.534204 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260811 03:08:01.534819 10885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
I20260811 03:08:01.536749 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.536311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46370)
0811 03:08:01.536518 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.536597 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260811 03:08:01.537118 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43943: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
I20260811 03:08:01.567621 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:43943
--webserver_interface=127.27.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:37453,127.27.109.61:43943,127.27.109.60:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:08:01.586634 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } attempt: 1
I20260811 03:08:01.588143 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.587518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46374)
0811 03:08:01.587795 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.587955 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260811 03:08:01.589375 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.588889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46384)
0811 03:08:01.589070 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.589176 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 03:08:01.589597 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43943: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
I20260811 03:08:01.657945 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } attempt: 2
I20260811 03:08:01.659437 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.658787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46396)
0811 03:08:01.659098 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.659225 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260811 03:08:01.660676 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.660201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46398)
0811 03:08:01.660388 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.660503 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260811 03:08:01.660913 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43943: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
I20260811 03:08:01.752207 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } attempt: 3
I20260811 03:08:01.753533 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.752927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46410)
0811 03:08:01.753184 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.753337 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260811 03:08:01.754805 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.754289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46416)
0811 03:08:01.754467 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.754563 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260811 03:08:01.755064 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43943: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
I20260811 03:08:01.898375 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } attempt: 4
I20260811 03:08:01.900036 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.899328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46418)
0811 03:08:01.899695 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.899816 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260811 03:08:01.901379 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:01.900860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46422)
0811 03:08:01.901028 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:01.901193 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 03:08:01.901644 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:43943: Network error: Client connection negotiation failed: client connection to 127.27.109.61:43943: connect: Connection refused (error 111)
W20260811 03:08:01.943977 10950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:01.944325 10950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:01.944511 10950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:01.953496 10950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:01.953637 10950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:01.953730 10950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:01.953816 10950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:01.953898 10950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:01.965929 10950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:37453,127.27.109.61:43943,127.27.109.60:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:43943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:01.968071 10950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:01.969713 10950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:01.985718 10955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:01.985679 10958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:01.985648 10956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:01.987253 10950 server_base.cc:1034] running on GCE node
I20260811 03:08:01.988248 10950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:01.989889 10950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:01.991132 10950 hybrid_clock.cc:648] HybridClock initialized: now 1786417681991046 us; error 72 us; skew 500 ppm
I20260811 03:08:01.991575 10950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:01 dist-test-slave-8tdl krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:01.995326 10950 init.cc:377] Logged in from keytab as kudu/127.27.109.61@KRBTEST.COM (short username kudu)
I20260811 03:08:01.997076 10950 webserver.cc:460] Webserver started at http://127.27.109.61:39163/ using document root <none> and password file <none>
I20260811 03:08:01.997843 10950 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:01.998011 10950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:01.998380 10950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:02.001839 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/data/instance:
uuid: "60f539d5a82f4a748722029c78f87b5a"
format_stamp: "Formatted at 2026-08-11 03:08:01 on dist-test-slave-8tdl"
server_key: "4291f3eb3318ec4001ab2a7501082405"
server_key_iv: "fe82055a73a3604571ad73ae956fea22"
server_key_version: "encryptionkey@0"
I20260811 03:08:02.002854 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/wal/instance:
uuid: "60f539d5a82f4a748722029c78f87b5a"
format_stamp: "Formatted at 2026-08-11 03:08:01 on dist-test-slave-8tdl"
server_key: "4291f3eb3318ec4001ab2a7501082405"
server_key_iv: "fe82055a73a3604571ad73ae956fea22"
server_key_version: "encryptionkey@0"
I20260811 03:08:02.009279 10950 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260811 03:08:02.015405 10965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:02.017562 10950 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260811 03:08:02.017830 10950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/wal
uuid: "60f539d5a82f4a748722029c78f87b5a"
format_stamp: "Formatted at 2026-08-11 03:08:01 on dist-test-slave-8tdl"
server_key: "4291f3eb3318ec4001ab2a7501082405"
server_key_iv: "fe82055a73a3604571ad73ae956fea22"
server_key_version: "encryptionkey@0"
I20260811 03:08:02.018127 10950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:02.046591 10950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:02.051383 10950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:02.051716 10950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:02.072919 10950 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:43943
I20260811 03:08:02.073057 11023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:43943 every 8 connection(s)
I20260811 03:08:02.074434 10950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
I20260811 03:08:02.078612 11024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:02.080623 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 10950
I20260811 03:08:02.080840 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-1/wal/instance
I20260811 03:08:02.081439 28084 external_mini_cluster.cc:1428] Setting key 68bbd9c11932c66a2b81005f2b220e2f
I20260811 03:08:02.089291 11024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.60@KRBTEST.COM" created.
I20260811 03:08:02.114823 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.093291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.27.109.62:37453)
0811 03:08:02.093966 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.093975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.095152 (+ 1177us) server_negotiation.cc:407] Connection header received
0811 03:08:02.097219 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.097225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.097323 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.097447 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.099234 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.100071 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.101730 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.102182 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.102323 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.106265 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.106273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.106279 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.106337 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.110612 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.111766 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.111771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.111774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.111940 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.112607 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.112612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.112614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.112980 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.113148 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.114155 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.114466 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":308,"threads_started":1}
I20260811 03:08:02.116204 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.094009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37453 (local address 127.0.0.1:52658)
0811 03:08:02.094973 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.095023 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:02.096894 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:02.097735 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:02.097745 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:02.098321 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:02.099033 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.099053 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.100267 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.100271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.101567 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.101582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.102438 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.102447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.102751 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.103491 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:02.103515 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:02.105909 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:02.110853 (+ 4944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.110865 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.110888 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.111586 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.112149 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.112157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.112163 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.112435 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.113188 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:02.113199 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:02.113647 (+ 448us) client_negotiation.cc:769] Sending connection context
0811 03:08:02.113951 (+ 304us) client_negotiation.cc:240] Negotiation successful
0811 03:08:02.114526 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":401,"threads_started":1}
I20260811 03:08:02.118207 11024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:08:02.138360 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.119209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46430)
0811 03:08:02.119499 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.119532 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:02.119707 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:02.120606 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:02.120610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:02.121044 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:02.121221 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.121230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.123392 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.123401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.124955 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.124975 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.126290 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.126298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.126642 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.127404 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:02.127433 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:02.129907 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:02.135343 (+ 5436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.135348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.135351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.135842 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.136403 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.136406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.136409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.136539 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.137462 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:02.137466 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:02.137592 (+ 126us) client_negotiation.cc:769] Sending connection context
0811 03:08:02.137718 (+ 126us) client_negotiation.cc:240] Negotiation successful
0811 03:08:02.138080 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 03:08:02.138933 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.119290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.27.109.61:43943)
0811 03:08:02.120010 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.120020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.120056 (+ 36us) server_negotiation.cc:407] Connection header received
0811 03:08:02.120121 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.120131 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.120419 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.120585 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.121402 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.123178 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.125154 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.126112 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.126255 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.130191 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.130219 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.130244 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.130302 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.135134 (+ 4832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.136046 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.136060 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.136067 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.136272 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.136648 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.136657 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.136664 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.137274 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.137551 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.137901 (+ 350us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.138271 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":505,"thread_start_us":325,"threads_started":1}
I20260811 03:08:02.142017 11024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:08:02.143656 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.142939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46670)
0811 03:08:02.143206 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.143423 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260811 03:08:02.143996 10969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.145963 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.145569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46686)
0811 03:08:02.145761 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.145847 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260811 03:08:02.146329 11024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
I20260811 03:08:02.159128 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:37219
--webserver_interface=127.27.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:37453,127.27.109.61:43943,127.27.109.60:37219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:08:02.195701 11024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 1
I20260811 03:08:02.197038 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.196508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46692)
0811 03:08:02.196747 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.196876 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260811 03:08:02.198104 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.197758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46700)
0811 03:08:02.197894 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.197977 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 03:08:02.198385 11024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.200913 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } attempt: 5
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:08:02.215911 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.201694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.27.109.61:43943)
0811 03:08:02.201851 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.201858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.201944 (+ 86us) server_negotiation.cc:407] Connection header received
0811 03:08:02.202234 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.202242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.202330 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.202449 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.203473 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.204181 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.205884 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.206409 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.206517 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.210145 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.210151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.210154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.210188 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.213000 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.213697 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.213704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.213707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.213832 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.214231 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.214237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.214241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.214546 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.214726 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.215298 (+ 572us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.215546 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 03:08:02.215909 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.201627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46436)
0811 03:08:02.201829 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.201850 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:02.202029 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:02.202515 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:02.202521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:02.203049 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:02.203302 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.203316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.204324 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.204329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.205709 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.205728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.206579 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.206585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.206805 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.207469 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:02.207493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:02.209929 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:02.213159 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.213165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.213170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.213572 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.213952 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.213958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.213962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.214121 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.214707 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:02.214713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:02.214926 (+ 213us) client_negotiation.cc:769] Sending connection context
0811 03:08:02.215165 (+ 239us) client_negotiation.cc:240] Negotiation successful
0811 03:08:02.215552 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 03:08:02.217315 10942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:08:02.218636 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.218045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46710)
0811 03:08:02.218277 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.218400 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 03:08:02.219791 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.219317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46726)
0811 03:08:02.219501 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.219638 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 03:08:02.220077 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.266647 11024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 2
I20260811 03:08:02.268055 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.267512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46732)
0811 03:08:02.267731 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.267832 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260811 03:08:02.269178 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.268790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46748)
0811 03:08:02.268943 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.269045 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 03:08:02.269465 11024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.271347 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 1
I20260811 03:08:02.272573 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.272039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46764)
0811 03:08:02.272254 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.272370 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260811 03:08:02.273629 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.273245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46768)
0811 03:08:02.273402 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.273490 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 03:08:02.273857 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.342115 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 2
I20260811 03:08:02.343566 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.342954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46778)
0811 03:08:02.343219 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.343333 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 03:08:02.344774 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.344365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46782)
0811 03:08:02.344571 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.344653 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260811 03:08:02.345086 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.360723 11024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 3
I20260811 03:08:02.362112 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.361505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46784)
0811 03:08:02.361752 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.361917 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 03:08:02.363442 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.362969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46788)
0811 03:08:02.363213 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.363304 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260811 03:08:02.363682 11024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.451335 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 3
I20260811 03:08:02.452673 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.452059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46792)
0811 03:08:02.452315 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.452437 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260811 03:08:02.453962 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.453483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46806)
0811 03:08:02.453709 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.453810 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260811 03:08:02.454247 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.506894 11024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 4
I20260811 03:08:02.508049 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.507535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46812)
0811 03:08:02.507813 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.507895 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260811 03:08:02.509064 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.508738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46820)
0811 03:08:02.508895 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.508960 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 03:08:02.509279 11024 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
W20260811 03:08:02.529824 11034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:02.530190 11034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:02.530347 11034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:02.538051 11034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:02.538235 11034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:02.538350 11034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:02.538462 11034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:02.538545 11034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:02.548789 11034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:37453,127.27.109.61:43943,127.27.109.60:37219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:37219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:02.550434 11034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:02.551959 11034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:02.565116 11040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:02.565191 11041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:02.565189 11043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:02.565768 11034 server_base.cc:1034] running on GCE node
I20260811 03:08:02.566757 11034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:02.568367 11034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:02.569638 11034 hybrid_clock.cc:648] HybridClock initialized: now 1786417682569573 us; error 71 us; skew 500 ppm
I20260811 03:08:02.570065 11034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:02.573833 11034 init.cc:377] Logged in from keytab as kudu/127.27.109.60@KRBTEST.COM (short username kudu)
I20260811 03:08:02.575516 11034 webserver.cc:460] Webserver started at http://127.27.109.60:40789/ using document root <none> and password file <none>
I20260811 03:08:02.576274 11034 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:02.576387 11034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:02.576673 11034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:02.580220 11034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/data/instance:
uuid: "40f90283de4a44c394a42f6651c295db"
format_stamp: "Formatted at 2026-08-11 03:08:02 on dist-test-slave-8tdl"
server_key: "90b563581abd5707b834a4f663c0f0d2"
server_key_iv: "1f456bbe2b485c66a289459ce3deb8e0"
server_key_version: "encryptionkey@0"
I20260811 03:08:02.581095 11034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/wal/instance:
uuid: "40f90283de4a44c394a42f6651c295db"
format_stamp: "Formatted at 2026-08-11 03:08:02 on dist-test-slave-8tdl"
server_key: "90b563581abd5707b834a4f663c0f0d2"
server_key_iv: "1f456bbe2b485c66a289459ce3deb8e0"
server_key_version: "encryptionkey@0"
I20260811 03:08:02.586891 11034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260811 03:08:02.590535 11050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:02.591984 11034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:08:02.592170 11034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/wal
uuid: "40f90283de4a44c394a42f6651c295db"
format_stamp: "Formatted at 2026-08-11 03:08:02 on dist-test-slave-8tdl"
server_key: "90b563581abd5707b834a4f663c0f0d2"
server_key_iv: "1f456bbe2b485c66a289459ce3deb8e0"
server_key_version: "encryptionkey@0"
I20260811 03:08:02.592401 11034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:02.609725 11034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:02.613834 11034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:02.614091 11034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:02.631497 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 4
I20260811 03:08:02.632736 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.632159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46830)
0811 03:08:02.632415 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.632534 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260811 03:08:02.633985 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.633506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46832)
0811 03:08:02.633709 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.633816 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 03:08:02.634228 10942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37219: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37219: connect: Connection refused (error 111)
I20260811 03:08:02.634474 11034 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:37219
I20260811 03:08:02.634487 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:37219 every 8 connection(s)
I20260811 03:08:02.635983 11034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
I20260811 03:08:02.639945 11108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:02.640570 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 11034
I20260811 03:08:02.640791 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786417454123933-28084-0/minicluster-data/master-2/wal/instance
I20260811 03:08:02.641412 28084 external_mini_cluster.cc:1428] Setting key ba9f497230977d2d921e8edc49eadaf8
I20260811 03:08:02.649211 11108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:02.667747 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.643477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.27.109.60:37219)
0811 03:08:02.644364 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.644376 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.644422 (+ 46us) server_negotiation.cc:407] Connection header received
0811 03:08:02.646648 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.646719 (+ 71us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.647150 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.647573 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.649182 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.650859 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.652599 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.653683 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.653879 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.658914 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.658925 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.658939 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.658991 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.662884 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.663933 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.663989 (+ 56us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.663996 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.664166 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.664654 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.664659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.664663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.665264 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.665571 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.666094 (+ 523us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.666590 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":390,"threads_started":1}
I20260811 03:08:02.669265 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.652002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:37453 (local address 127.0.0.1:52666)
0811 03:08:02.653199 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.653246 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:02.653413 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:02.653819 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:02.653826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:02.654769 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:02.655077 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.655094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.656663 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.656667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.657773 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.657790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.658914 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.658923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.659155 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.659863 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:02.659902 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:02.662975 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:02.666305 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.666318 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.666327 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.666937 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.667347 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.667352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.667356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.667500 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.668164 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:02.668174 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:02.668363 (+ 189us) client_negotiation.cc:769] Sending connection context
0811 03:08:02.668517 (+ 154us) client_negotiation.cc:240] Negotiation successful
0811 03:08:02.668814 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":769,"threads_started":1}
I20260811 03:08:02.669385 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.651986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.27.109.62:37453)
0811 03:08:02.652630 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.652640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.653358 (+ 718us) server_negotiation.cc:407] Connection header received
0811 03:08:02.653555 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.653559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.653627 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.653720 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.655278 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.656477 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.657921 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.658611 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.658767 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.663203 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.663213 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.663220 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.663273 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.666154 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.667064 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.667072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.667077 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.667229 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.667623 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.667627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.667629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.667973 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.668149 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.668683 (+ 534us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.668942 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":278,"threads_started":1}
I20260811 03:08:02.671205 11108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:02.686322 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.672020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:43943 (local address 127.0.0.1:46448)
0811 03:08:02.672401 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.672416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:02.672514 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:02.672862 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:02.672865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:02.673224 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:02.673383 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.673391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.674438 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.674443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.675392 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.675404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.676182 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.676188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.676385 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.677127 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:02.677145 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:02.679563 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:02.683634 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.683638 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.683641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.684118 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.684542 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.684546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.684549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.684731 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.685377 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:02.685380 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:02.685600 (+ 220us) client_negotiation.cc:769] Sending connection context
0811 03:08:02.685772 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 03:08:02.686065 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285}
I20260811 03:08:02.686724 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.672110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.27.109.61:43943)
0811 03:08:02.672282 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.672286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.672504 (+ 218us) server_negotiation.cc:407] Connection header received
0811 03:08:02.672615 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.672620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.672713 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.672817 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.673554 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.674264 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.675545 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.676012 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.676142 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.679787 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.679795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.679800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.679851 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.683479 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.684246 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.684252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.684259 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.684425 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.684852 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.684855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.684857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.685250 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.685403 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.685965 (+ 562us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.686221 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 03:08:02.687963 11108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:08:02.688050 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.671549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.27.109.62:37453)
0811 03:08:02.671729 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.671735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.671753 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:08:02.671858 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.671865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.671975 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.672081 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.673578 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.674265 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.675617 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.676020 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.676158 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.680811 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.680820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.680827 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.680889 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.684357 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.685220 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.685226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.685229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.685385 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.685862 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.685868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.685871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.686285 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.686616 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.687390 (+ 774us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.687687 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417682, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:08:02.703040 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.689051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.27.109.60:37219)
0811 03:08:02.689192 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.689195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.689208 (+ 13us) server_negotiation.cc:407] Connection header received
0811 03:08:02.689242 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.689245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.689292 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.689390 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.690269 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.691613 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.692966 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.693519 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.693661 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.697325 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.697334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.697340 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.697382 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.700491 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.701142 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.701147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.701151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.701267 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.701613 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.701620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.701624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.701916 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.702074 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.702407 (+ 333us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.702641 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 03:08:02.703361 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.688673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46858)
0811 03:08:02.688824 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.688846 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:02.688981 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:02.689406 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:02.689410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:02.689903 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:02.690091 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.690102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.691760 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.691764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.692826 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.692840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.693693 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.693697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.693880 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.694513 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:02.694536 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:02.697085 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:02.700640 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.700645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.700649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.701013 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.701374 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.701379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.701383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.701511 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.702045 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:02.702050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:02.702196 (+ 146us) client_negotiation.cc:769] Sending connection context
0811 03:08:02.702331 (+ 135us) client_negotiation.cc:240] Negotiation successful
0811 03:08:02.702849 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":148}
I20260811 03:08:02.706669 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.689822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.27.109.61:43943)
0811 03:08:02.689994 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.690001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.690063 (+ 62us) server_negotiation.cc:407] Connection header received
0811 03:08:02.690213 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.690219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.690290 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.690358 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.692175 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.693125 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.694543 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.695259 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.695393 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.700645 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.700651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.700654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.700692 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.703804 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.704579 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.704584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.704586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.704776 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.705146 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.705153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.705156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.705569 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.705854 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.706046 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.706295 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 03:08:02.709602 11108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: Bootstrap starting.
I20260811 03:08:02.712160 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:08:02.713526 11108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:02.714996 11108 log.cc:824] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:02.718565 11108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: No bootstrap required, opened a new log
I20260811 03:08:02.730530 11108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:02.731092 11108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:02.731243 11108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f90283de4a44c394a42f6651c295db, State: Initialized, Role: FOLLOWER
I20260811 03:08:02.732144 11108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:02.733713 11126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:02.734192 11126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:08:02.736383 11108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:08:02.739691 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.718598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.27.109.61:43943)
0811 03:08:02.718841 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.718847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.719442 (+ 595us) server_negotiation.cc:407] Connection header received
0811 03:08:02.719788 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.719795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.719882 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.719959 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.722412 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.723560 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.725923 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.726599 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.726733 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.730967 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.730973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.730977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.731025 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.734967 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.736238 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.736245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.736248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.736410 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.737157 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.737162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.737166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.737591 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.737825 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.739024 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.739295 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260811 03:08:02.743331 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.718602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.27.109.62:37453)
0811 03:08:02.718858 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.718864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.719862 (+ 998us) server_negotiation.cc:407] Connection header received
0811 03:08:02.720152 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.720158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.720253 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.720354 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.723509 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.725022 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.728209 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.729204 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.729337 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.734617 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.734625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.734629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.734706 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.738572 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.740214 (+ 1642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.740223 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.740229 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.740449 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.741048 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.741056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.741062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.741551 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.741857 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.742549 (+ 692us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.742891 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260811 03:08:02.753741 11137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:08:02.754006 11137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:08:02.754451 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.719937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46870 (local address 127.27.109.60:37219)
0811 03:08:02.720197 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.720203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.733359 (+ 13156us) server_negotiation.cc:407] Connection header received
0811 03:08:02.733681 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.733688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.733784 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.733864 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.735999 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.737215 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.739223 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.739992 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.740125 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.743900 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.743905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.743913 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.743947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.748109 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.749096 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.749101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.749104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.751077 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.751648 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.751656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.751662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.752145 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.753505 (+ 1360us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.753669 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.753938 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260811 03:08:02.749187 11034 master_runner.cc:186] Error getting master registration for 127.27.109.62:37453: OK, instance_id { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" instance_seqno: 1786417681484948 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 03:08:02.759487 11034 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 03:08:02.808503 11024 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 5
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260811 03:08:02.823459 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.809210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46886)
0811 03:08:02.809397 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.809422 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:02.809516 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:02.809873 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:02.809877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:02.810307 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:02.810474 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.810483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.812046 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.812050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.813301 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.813319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.814394 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.814400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.814640 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.815364 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:02.815383 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:02.817568 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:02.820831 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.820838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.820842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.821248 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.821661 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.821666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.821670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.821812 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.822377 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:02.822382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:02.822543 (+ 161us) client_negotiation.cc:769] Sending connection context
0811 03:08:02.822715 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 03:08:02.823038 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 03:08:02.823444 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.809298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.27.109.60:37219)
0811 03:08:02.809494 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.809500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.809522 (+ 22us) server_negotiation.cc:407] Connection header received
0811 03:08:02.809644 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.809650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.809729 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.809853 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.810638 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.811906 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.813494 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.814218 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.814389 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.817762 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.817767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.817770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.817806 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.820698 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.821371 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.821377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.821380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.821540 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.821925 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.821931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.821934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.822250 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.822412 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.822833 (+ 421us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.823089 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 03:08:02.828745 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: Bootstrap starting.
I20260811 03:08:02.831941 11024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:02.833009 11024 log.cc:824] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:02.836199 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: No bootstrap required, opened a new log
I20260811 03:08:02.845674 11024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:02.846091 11024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:02.846236 11024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f539d5a82f4a748722029c78f87b5a, State: Initialized, Role: FOLLOWER
I20260811 03:08:02.847004 11024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:02.848063 11141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:02.848451 11141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:08:02.849323 11024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [sys.catalog]: configured and running, proceeding with master startup.
W20260811 03:08:02.861686 11153 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:08:02.861907 11153 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260811 03:08:02.866335 11057 tablet.cc:2369] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:02.859193 10950 master_runner.cc:186] Error getting master registration for 127.27.109.62:37453: OK, instance_id { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" instance_seqno: 1786417681484948 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 03:08:02.874219 10950 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 03:08:02.911477 10942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } attempt: 5
Aug 11 03:08:02 dist-test-slave-8tdl krb5kdc[10853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417681, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260811 03:08:02.930963 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.912226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37219 (local address 127.0.0.1:46896)
0811 03:08:02.912519 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:02.912542 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:02.912651 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:02.913076 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:02.913082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:02.913633 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:02.913858 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.913872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.915573 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.915579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.917180 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:02.917197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.918388 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.918394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:02.918586 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.919371 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:02.919397 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:02.922465 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:02.928321 (+ 5856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.928327 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.928331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.928754 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.929135 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:02.929140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:02.929144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:02.929308 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:02.929874 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:02.929879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:02.930038 (+ 159us) client_negotiation.cc:769] Sending connection context
0811 03:08:02.930180 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 03:08:02.930487 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260811 03:08:02.930966 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:02.912337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.27.109.60:37219)
0811 03:08:02.912554 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:02.912561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:02.912612 (+ 51us) server_negotiation.cc:407] Connection header received
0811 03:08:02.912797 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:02.912803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:02.912891 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:02.913020 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:02.914058 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.915385 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.917352 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:02.918171 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:02.918318 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:02.922761 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:02.922772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:02.922778 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:02.922824 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:02.928166 (+ 5342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.928877 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.928885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.928890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.929020 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:02.929419 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:02.929424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:02.929428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:02.929736 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:02.929898 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:02.930275 (+ 377us) server_negotiation.cc:299] Negotiation successful
0811 03:08:02.930526 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:08:02.936072 10942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216: Bootstrap starting.
I20260811 03:08:02.939731 10942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:02.940861 10942 log.cc:824] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:02.943940 10942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216: No bootstrap required, opened a new log
I20260811 03:08:02.954435 10942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:02.954873 10942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:02.955024 10942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f22684fde4c460eba922a5bb62a5216, State: Initialized, Role: FOLLOWER
I20260811 03:08:02.955679 10942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:02.956727 11156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:02.957209 11156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:08:02.957981 10942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 03:08:02.975584 10891 tablet.cc:2369] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:08:02.979564 11169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:08:02.979719 11169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:08:02.985523 10866 master_runner.cc:263] No leader master found from master 1f22684fde4c460eba922a5bb62a5216
I20260811 03:08:02.985697 10866 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 03:08:03.031095 11156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:08:03.031517 11156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:03.033568 11156 leader_election.cc:288] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60f539d5a82f4a748722029c78f87b5a (127.27.109.61:43943), 40f90283de4a44c394a42f6651c295db (127.27.109.60:37219)
I20260811 03:08:03.034937 10996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f22684fde4c460eba922a5bb62a5216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f539d5a82f4a748722029c78f87b5a" is_pre_election: true
I20260811 03:08:03.035673 10996 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f22684fde4c460eba922a5bb62a5216 in term 0.
I20260811 03:08:03.036461 10885 leader_election.cc:302] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f22684fde4c460eba922a5bb62a5216, 60f539d5a82f4a748722029c78f87b5a; no voters:
I20260811 03:08:03.037818 11080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f22684fde4c460eba922a5bb62a5216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40f90283de4a44c394a42f6651c295db" is_pre_election: true
I20260811 03:08:03.038532 11080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f22684fde4c460eba922a5bb62a5216 in term 0.
I20260811 03:08:03.039623 11156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:08:03.039806 11156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:08:03.039947 11156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:03.043979 11156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:03.050117 10996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f22684fde4c460eba922a5bb62a5216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f539d5a82f4a748722029c78f87b5a"
I20260811 03:08:03.050406 10996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [term 0 FOLLOWER]: Advancing to term 1
W20260811 03:08:03.054723 10971 tablet.cc:2369] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:03.056591 10996 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f22684fde4c460eba922a5bb62a5216 in term 1.
I20260811 03:08:03.057212 10885 leader_election.cc:302] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f22684fde4c460eba922a5bb62a5216, 60f539d5a82f4a748722029c78f87b5a; no voters:
I20260811 03:08:03.061095 11080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f22684fde4c460eba922a5bb62a5216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40f90283de4a44c394a42f6651c295db"
I20260811 03:08:03.061395 11080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:03.065403 11080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f22684fde4c460eba922a5bb62a5216 in term 1.
I20260811 03:08:03.066349 11156 leader_election.cc:288] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [CANDIDATE]: Term 1 election: Requested vote from peers 60f539d5a82f4a748722029c78f87b5a (127.27.109.61:43943), 40f90283de4a44c394a42f6651c295db (127.27.109.60:37219)
I20260811 03:08:03.066658 11156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:08:03.066891 11156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [term 1 LEADER]: Becoming Leader. State: Replica: 1f22684fde4c460eba922a5bb62a5216, State: Running, Role: LEADER
I20260811 03:08:03.067569 11156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } }
I20260811 03:08:03.087936 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f22684fde4c460eba922a5bb62a5216. Latest consensus state: current_term: 1 leader_uuid: "1f22684fde4c460eba922a5bb62a5216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:03.088547 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:03.096369 11173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:08:03.099159 11173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:08:03.121428 11080 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [term 1 FOLLOWER]: Refusing update from remote peer 1f22684fde4c460eba922a5bb62a5216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:03.122665 11171 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:03.130475 10996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [term 1 FOLLOWER]: Refusing update from remote peer 1f22684fde4c460eba922a5bb62a5216: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:03.131625 11171 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:03.155784 11141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f22684fde4c460eba922a5bb62a5216. Latest consensus state: current_term: 1 leader_uuid: "1f22684fde4c460eba922a5bb62a5216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:03.156184 11141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:08:03.166311 11174 mvcc.cc:205] Tried to move back new op lower bound from 7317166830029008896 to 7317166829863186432. Current Snapshot: MvccSnapshot[applied={T|T < 7317166830029008896}]
I20260811 03:08:03.168725 11176 mvcc.cc:205] Tried to move back new op lower bound from 7317166830029008896 to 7317166829863186432. Current Snapshot: MvccSnapshot[applied={T|T < 7317166830029008896}]
I20260811 03:08:03.176041 11126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f22684fde4c460eba922a5bb62a5216. Latest consensus state: current_term: 1 leader_uuid: "1f22684fde4c460eba922a5bb62a5216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:03.176430 11126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:08:03.176462 11156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f22684fde4c460eba922a5bb62a5216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:03.176826 11156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:03.178928 11156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f22684fde4c460eba922a5bb62a5216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:03.179746 11175 mvcc.cc:205] Tried to move back new op lower bound from 7317166830029008896 to 7317166829863186432. Current Snapshot: MvccSnapshot[applied={T|T < 7317166830029008896}]
I20260811 03:08:03.179746 11156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:03.185001 11141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f22684fde4c460eba922a5bb62a5216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:03.185304 11141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:08:03.185582 11126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f22684fde4c460eba922a5bb62a5216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f22684fde4c460eba922a5bb62a5216" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37453 } } peers { permanent_uuid: "60f539d5a82f4a748722029c78f87b5a" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 43943 } } peers { permanent_uuid: "40f90283de4a44c394a42f6651c295db" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37219 } } }
I20260811 03:08:03.185851 11126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:08:03.187541 11173 catalog_manager.cc:1347] Generated new cluster ID: e7cea7515d0e4a728200ccd6524598d3
I20260811 03:08:03.187656 11173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:08:03.242142 11173 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:08:03.244369 11173 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:08:03.274708 11173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f22684fde4c460eba922a5bb62a5216: Generated new TSK 0
I20260811 03:08:03.276072 11173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:08:03.756237 11137 catalog_manager.cc:1259] Loaded cluster ID: e7cea7515d0e4a728200ccd6524598d3
I20260811 03:08:03.756349 11137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: loading cluster ID for follower catalog manager: success
I20260811 03:08:03.759097 11137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: acquiring CA information for follower catalog manager: success
I20260811 03:08:03.761135 11137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 40f90283de4a44c394a42f6651c295db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 03:08:03.864658 11153 catalog_manager.cc:1259] Loaded cluster ID: e7cea7515d0e4a728200ccd6524598d3
I20260811 03:08:03.864791 11153 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: loading cluster ID for follower catalog manager: success
I20260811 03:08:03.867746 11153 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: acquiring CA information for follower catalog manager: success
I20260811 03:08:03.869658 11153 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60f539d5a82f4a748722029c78f87b5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 03:08:03.879959 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 10866
I20260811 03:08:03.881152 10935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:03.995023 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 10866
I20260811 03:08:04.026647 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 10950
I20260811 03:08:04.027997 11019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:04.145006 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 10950
I20260811 03:08:04.172151 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 11034
I20260811 03:08:04.173534 11102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:04.286559 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 11034
2026-08-11T03:08:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4809 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:08:04 dist-test-slave-8tdl krb5kdc[11193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:08:04 dist-test-slave-8tdl krb5kdc[11193](info): set up 2 sockets
Aug 11 03:08:04 dist-test-slave-8tdl krb5kdc[11193](info): commencing operation
krb5kdc: starting...
W20260811 03:08:05.541749 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.001s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:08:05 dist-test-slave-8tdl krb5kdc[11193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:08:05.868373 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41005
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:06.234827 11208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:06.235253 11208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:06.235414 11208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:06.244127 11208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:06.244310 11208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:06.244439 11208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:06.244546 11208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:06.244637 11208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:06.255833 11208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:06.257380 11208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:06.258869 11208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:06.271343 11213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:06.271309 11216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:06.271401 11214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:06.272339 11208 server_base.cc:1034] running on GCE node
I20260811 03:08:06.273191 11208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:06.274807 11208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:06.276046 11208 hybrid_clock.cc:648] HybridClock initialized: now 1786417686275998 us; error 71 us; skew 500 ppm
I20260811 03:08:06.276450 11208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:06 dist-test-slave-8tdl krb5kdc[11193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417686, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:06.280171 11208 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:08:06.281906 11208 webserver.cc:460] Webserver started at http://127.27.109.62:35075/ using document root <none> and password file <none>
I20260811 03:08:06.282655 11208 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:06.282860 11208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:06.283253 11208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:06.286502 11208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "3e096dc296d74a8d9498851504b4cb68"
format_stamp: "Formatted at 2026-08-11 03:08:06 on dist-test-slave-8tdl"
server_key: "9b799647cec3d6bd0ad02c1d64491f57"
server_key_iv: "a0ae95b73b3b149468f8016ed3b81468"
server_key_version: "encryptionkey@0"
I20260811 03:08:06.287388 11208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "3e096dc296d74a8d9498851504b4cb68"
format_stamp: "Formatted at 2026-08-11 03:08:06 on dist-test-slave-8tdl"
server_key: "9b799647cec3d6bd0ad02c1d64491f57"
server_key_iv: "a0ae95b73b3b149468f8016ed3b81468"
server_key_version: "encryptionkey@0"
I20260811 03:08:06.293073 11208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260811 03:08:06.297016 11226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:06.298554 11208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:08:06.298791 11208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "3e096dc296d74a8d9498851504b4cb68"
format_stamp: "Formatted at 2026-08-11 03:08:06 on dist-test-slave-8tdl"
server_key: "9b799647cec3d6bd0ad02c1d64491f57"
server_key_iv: "a0ae95b73b3b149468f8016ed3b81468"
server_key_version: "encryptionkey@0"
I20260811 03:08:06.298993 11208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:06.334210 11208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:06.339473 11208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:06.339830 11208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:06.361552 11208 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41005
I20260811 03:08:06.361585 11286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41005 every 8 connection(s)
I20260811 03:08:06.363296 11208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:08:06.367339 11287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:06.371019 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 11208
I20260811 03:08:06.371282 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:08:06.372023 28084 external_mini_cluster.cc:1428] Setting key b153bc6de4e9fc9720fa06374e63357d
I20260811 03:08:06.381353 11287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68: Bootstrap starting.
I20260811 03:08:06.384847 11287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:06.386000 11287 log.cc:824] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68: Log is configured to *not* fsync() on all Append() calls
Aug 11 03:08:06 dist-test-slave-8tdl krb5kdc[11193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:06.389532 11287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68: No bootstrap required, opened a new log
I20260811 03:08:06.400058 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:06.374418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.27.109.62:41005)
0811 03:08:06.375397 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:06.375410 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:06.375455 (+ 45us) server_negotiation.cc:407] Connection header received
0811 03:08:06.377330 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:06.377364 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:06.377940 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:06.378398 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:06.380118 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:06.381615 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:06.383239 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:06.384013 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:06.384244 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:06.389145 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:06.389157 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:06.389185 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:06.389244 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:06.394045 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:06.395086 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:06.395098 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:06.395109 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:06.395390 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:06.395863 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:06.395871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:06.395876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:06.396820 (+ 944us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:06.397219 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:06.397653 (+ 434us) server_negotiation.cc:299] Negotiation successful
0811 03:08:06.398227 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":479,"threads_started":1}
I20260811 03:08:06.402653 11287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e096dc296d74a8d9498851504b4cb68" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41005 } }
I20260811 03:08:06.403275 11287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:06.403431 11287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e096dc296d74a8d9498851504b4cb68, State: Initialized, Role: FOLLOWER
I20260811 03:08:06.404160 11287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e096dc296d74a8d9498851504b4cb68" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41005 } }
I20260811 03:08:06.404532 11287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:08:06.404709 11287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:08:06.404917 11287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:06.405310 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:08:06.407727 11287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e096dc296d74a8d9498851504b4cb68" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41005 } }
I20260811 03:08:06.408484 11287 leader_election.cc:302] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e096dc296d74a8d9498851504b4cb68; no voters:
I20260811 03:08:06.409415 11287 leader_election.cc:288] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:08:06.409528 11293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:08:06.410827 11293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [term 1 LEADER]: Becoming Leader. State: Replica: 3e096dc296d74a8d9498851504b4cb68, State: Running, Role: LEADER
I20260811 03:08:06.411407 11293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e096dc296d74a8d9498851504b4cb68" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41005 } }
I20260811 03:08:06.411718 11287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:08:06.414285 11295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e096dc296d74a8d9498851504b4cb68. Latest consensus state: current_term: 1 leader_uuid: "3e096dc296d74a8d9498851504b4cb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e096dc296d74a8d9498851504b4cb68" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41005 } } }
I20260811 03:08:06.415102 11295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:06.415045 11294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e096dc296d74a8d9498851504b4cb68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e096dc296d74a8d9498851504b4cb68" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41005 } } }
I20260811 03:08:06.415364 11294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:06.417443 11302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:08:06.423029 11302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:08:06.433173 11302 catalog_manager.cc:1347] Generated new cluster ID: 8e5026a0c94c47d381e5c9533ab88fea
I20260811 03:08:06.433312 11302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:08:06.472249 11302 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:08:06.473658 11302 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:08:06.481527 11302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e096dc296d74a8d9498851504b4cb68: Generated new TSK 0
I20260811 03:08:06.482420 11302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:08:06.519032 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 11208
I20260811 03:08:06.520334 11281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:06.620262 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 11208
2026-08-11T03:08:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2329 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:08:06 dist-test-slave-8tdl krb5kdc[11316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:08:06 dist-test-slave-8tdl krb5kdc[11316](info): set up 2 sockets
Aug 11 03:08:06 dist-test-slave-8tdl krb5kdc[11316](info): commencing operation
krb5kdc: starting...
W20260811 03:08:07.844656 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.003s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:08:08 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:08:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:08:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:08:08.171061 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32863
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:32863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:08.541358 11331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:08.541702 11331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:08.541798 11331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:08.549486 11331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:08.549625 11331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:08.549675 11331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:08.549734 11331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:08.549780 11331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:08.560636 11331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:32863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:08.562108 11331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:08.563588 11331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:08.575846 11339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:08.575791 11336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:08.575904 11337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:08.577121 11331 server_base.cc:1034] running on GCE node
I20260811 03:08:08.577996 11331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:08.579610 11331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:08.580835 11331 hybrid_clock.cc:648] HybridClock initialized: now 1786417688580761 us; error 53 us; skew 500 ppm
I20260811 03:08:08.581207 11331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:08 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417688, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:08.584637 11331 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:08:08.586167 11331 webserver.cc:460] Webserver started at http://127.27.109.62:43105/ using document root <none> and password file <none>
I20260811 03:08:08.586980 11331 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:08.587131 11331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:08.587457 11331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:08.590636 11331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "836425ae2dee4a99b606933df3f7e0a6"
format_stamp: "Formatted at 2026-08-11 03:08:08 on dist-test-slave-8tdl"
server_key: "2460ed5a69e453d623a4f6c1421fe905"
server_key_iv: "100b93d8b89b9dd925c82d42fbc8f234"
server_key_version: "encryptionkey@0"
I20260811 03:08:08.591563 11331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "836425ae2dee4a99b606933df3f7e0a6"
format_stamp: "Formatted at 2026-08-11 03:08:08 on dist-test-slave-8tdl"
server_key: "2460ed5a69e453d623a4f6c1421fe905"
server_key_iv: "100b93d8b89b9dd925c82d42fbc8f234"
server_key_version: "encryptionkey@0"
I20260811 03:08:08.597429 11331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:08:08.601555 11350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:08.603245 11331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 03:08:08.603477 11331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "836425ae2dee4a99b606933df3f7e0a6"
format_stamp: "Formatted at 2026-08-11 03:08:08 on dist-test-slave-8tdl"
server_key: "2460ed5a69e453d623a4f6c1421fe905"
server_key_iv: "100b93d8b89b9dd925c82d42fbc8f234"
server_key_version: "encryptionkey@0"
I20260811 03:08:08.603713 11331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:08.617513 11331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:08.622200 11331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:08.622524 11331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:08.643409 11331 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:32863
I20260811 03:08:08.643456 11410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:32863 every 8 connection(s)
I20260811 03:08:08.645040 11331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:08:08.649073 11411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:08.655256 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 11331
I20260811 03:08:08.655507 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:08:08.656219 28084 external_mini_cluster.cc:1428] Setting key 0e4ac77043ce79fc098edceb6835c32f
I20260811 03:08:08.662087 11411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Bootstrap starting.
I20260811 03:08:08.666347 11411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:08.667732 11411 log.cc:824] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:08.671541 11411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: No bootstrap required, opened a new log
Aug 11 03:08:08 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:08.682253 11411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:08.682849 11411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:08.682967 11411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836425ae2dee4a99b606933df3f7e0a6, State: Initialized, Role: FOLLOWER
I20260811 03:08:08.683708 11411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:08.684021 11411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:08:08.684178 11411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:08:08.684374 11411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:08.684718 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:08.658578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.27.109.62:32863)
0811 03:08:08.659628 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:08.659646 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:08.659700 (+ 54us) server_negotiation.cc:407] Connection header received
0811 03:08:08.662208 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:08.662240 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:08.662829 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:08.663318 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:08.664860 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:08.667064 (+ 2204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:08.668889 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:08.669969 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:08.670197 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:08.674627 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:08.674640 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:08.674668 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:08.674754 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:08.679484 (+ 4730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:08.680360 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:08.680376 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:08.680388 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:08.680640 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:08.681054 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:08.681064 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:08.681067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:08.681855 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:08.682192 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:08.682580 (+ 388us) server_negotiation.cc:299] Negotiation successful
0811 03:08:08.683147 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":459,"threads_started":1}
I20260811 03:08:08.687214 11411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:08.687814 11411 leader_election.cc:302] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 836425ae2dee4a99b606933df3f7e0a6; no voters:
I20260811 03:08:08.688472 11411 leader_election.cc:288] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:08:08.688628 11416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:08:08.690002 11416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 1 LEADER]: Becoming Leader. State: Replica: 836425ae2dee4a99b606933df3f7e0a6, State: Running, Role: LEADER
I20260811 03:08:08.690753 11416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:08.690977 11411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:08:08.693666 11418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836425ae2dee4a99b606933df3f7e0a6. Latest consensus state: current_term: 1 leader_uuid: "836425ae2dee4a99b606933df3f7e0a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } } }
I20260811 03:08:08.694110 11418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:08.694777 11417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "836425ae2dee4a99b606933df3f7e0a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } } }
I20260811 03:08:08.695259 11417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:08.697072 11424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:08:08.703481 11424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:08:08.713397 11424 catalog_manager.cc:1347] Generated new cluster ID: 9b130beb2c3a402da9b24be0af2828cd
I20260811 03:08:08.713546 11424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:08:08.726804 11424 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:08:08.728286 11424 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:08:08.740002 11424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Generated new TSK 0
I20260811 03:08:08.740991 11424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:08:08.815639 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32863
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:09.204701 11441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:09.205081 11441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:09.205288 11441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:09.213828 11441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:09.213969 11441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:09.214155 11441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:08:09.224553 11441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:09.225987 11441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:09.227435 11441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:09.241163 11447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:09.241357 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:09.241614 11448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:09.244318 11441 server_base.cc:1034] running on GCE node
I20260811 03:08:09.245115 11441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:09.246196 11441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:09.247411 11441 hybrid_clock.cc:648] HybridClock initialized: now 1786417689247361 us; error 40 us; skew 500 ppm
I20260811 03:08:09.247772 11441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:09 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417689, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:09.251372 11441 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:08:09.253093 11441 webserver.cc:460] Webserver started at http://127.27.109.1:44451/ using document root <none> and password file <none>
I20260811 03:08:09.253861 11441 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:09.254005 11441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:09.254338 11441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:09.257529 11441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "def07bb84f0345a1baffd3df05e63670"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "39b20b899cd49795c10d4498a423f1a1"
server_key_iv: "1eacb5ea17329b23f83d55711a61e92c"
server_key_version: "encryptionkey@0"
I20260811 03:08:09.258376 11441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "def07bb84f0345a1baffd3df05e63670"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "39b20b899cd49795c10d4498a423f1a1"
server_key_iv: "1eacb5ea17329b23f83d55711a61e92c"
server_key_version: "encryptionkey@0"
I20260811 03:08:09.264037 11441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 03:08:09.267935 11457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:09.269529 11441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:08:09.269714 11441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "def07bb84f0345a1baffd3df05e63670"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "39b20b899cd49795c10d4498a423f1a1"
server_key_iv: "1eacb5ea17329b23f83d55711a61e92c"
server_key_version: "encryptionkey@0"
I20260811 03:08:09.269929 11441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:09.299007 11441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:09.304023 11441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:09.304364 11441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:09.305449 11441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:09.307075 11441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:09.307164 11441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:09.307299 11441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:09.307363 11441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:09.344806 11441 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:43711
I20260811 03:08:09.344858 11582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:43711 every 8 connection(s)
I20260811 03:08:09.346503 11441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:08:09.351055 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 11441
I20260811 03:08:09.351318 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:08:09.352035 28084 external_mini_cluster.cc:1428] Setting key 139821a3b6febdbfeb276eb28e09db8b
Aug 11 03:08:09 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417689, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:09.368793 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:09.349411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:52817 (local address 127.27.109.62:32863)
0811 03:08:09.349954 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:09.349961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:09.350870 (+ 909us) server_negotiation.cc:407] Connection header received
0811 03:08:09.353176 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:09.353182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:09.353278 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:09.353405 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:09.355343 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:09.356129 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:09.357790 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:09.358223 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:09.358375 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:09.362130 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:09.362135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:09.362138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:09.362175 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:09.364986 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:09.366057 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:09.366063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:09.366065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:09.366211 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:09.366822 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:09.366826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:09.366828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:09.367154 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:09.367310 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:09.368279 (+ 969us) server_negotiation.cc:299] Negotiation successful
0811 03:08:09.368513 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260811 03:08:09.370596 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:09.349824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:52817)
0811 03:08:09.350643 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:09.350731 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:09.352817 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:09.353674 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:09.353683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:09.354408 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:09.355152 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:09.355172 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:09.356304 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:09.356308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:09.357610 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:09.357630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:09.358478 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:09.358487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:09.358803 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:09.359561 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:09.359593 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:09.361870 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:09.365166 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:09.365176 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:09.365197 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:09.365826 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:09.366339 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:09.366346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:09.366351 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:09.366647 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:09.367333 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:09.367345 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:09.367844 (+ 499us) client_negotiation.cc:769] Sending connection context
0811 03:08:09.368165 (+ 321us) client_negotiation.cc:240] Negotiation successful
0811 03:08:09.368820 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":309,"threads_started":1}
I20260811 03:08:09.372426 11583 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32863
I20260811 03:08:09.372781 11583 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:09.373657 11583 heartbeater.cc:511] Master 127.27.109.62:32863 requested a full tablet report, sending...
I20260811 03:08:09.376008 11370 ts_manager.cc:194] Registered new tserver with Master: def07bb84f0345a1baffd3df05e63670 (127.27.109.1:43711)
I20260811 03:08:09.377624 11370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:52817
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:08:09.422102 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32863
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:09.783595 11590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:09.783960 11590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:09.784137 11590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:09.792032 11590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:09.792214 11590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:09.792428 11590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:08:09.803491 11590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:09.804975 11590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:09.806420 11590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:09.820756 11595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:09.820796 11596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:09.820845 11598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:09.821887 11590 server_base.cc:1034] running on GCE node
I20260811 03:08:09.822576 11590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:09.823652 11590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:09.824939 11590 hybrid_clock.cc:648] HybridClock initialized: now 1786417689824908 us; error 69 us; skew 500 ppm
I20260811 03:08:09.825408 11590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:09 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417689, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:09.829151 11590 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:08:09.831070 11590 webserver.cc:460] Webserver started at http://127.27.109.2:38303/ using document root <none> and password file <none>
I20260811 03:08:09.831857 11590 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:09.832029 11590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:09.832397 11590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:09.836120 11590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "900519cb725642278519e3032efa2a6e"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "2933851f6e3be2a826c85d05ae441376"
server_key_iv: "2d60a2c94eb760022cb5d5ff243e4da5"
server_key_version: "encryptionkey@0"
I20260811 03:08:09.837036 11590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "900519cb725642278519e3032efa2a6e"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "2933851f6e3be2a826c85d05ae441376"
server_key_iv: "2d60a2c94eb760022cb5d5ff243e4da5"
server_key_version: "encryptionkey@0"
I20260811 03:08:09.842877 11590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260811 03:08:09.846881 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:09.848415 11590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:08:09.848640 11590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "900519cb725642278519e3032efa2a6e"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "2933851f6e3be2a826c85d05ae441376"
server_key_iv: "2d60a2c94eb760022cb5d5ff243e4da5"
server_key_version: "encryptionkey@0"
I20260811 03:08:09.848899 11590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:09.873152 11590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:09.878501 11590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:09.878912 11590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:09.880147 11590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:09.881733 11590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:09.881852 11590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:09.882021 11590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:09.882103 11590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:09.921101 11590 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45895
I20260811 03:08:09.921146 11728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45895 every 8 connection(s)
I20260811 03:08:09.922607 11590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:08:09.924319 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 11590
I20260811 03:08:09.924532 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:08:09.925032 28084 external_mini_cluster.cc:1428] Setting key 0319af354411c8820ce2772f846e395c
Aug 11 03:08:09 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417689, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:09.944780 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:09.925445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:40293 (local address 127.27.109.62:32863)
0811 03:08:09.926441 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:09.926448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:09.929026 (+ 2578us) server_negotiation.cc:407] Connection header received
0811 03:08:09.929083 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:09.929087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:09.929151 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:09.929275 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:09.930916 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:09.931644 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:09.933236 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:09.933696 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:09.933847 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:09.938111 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:09.938117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:09.938120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:09.938157 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:09.941277 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:09.942161 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:09.942166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:09.942168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:09.942306 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:09.942854 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:09.942858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:09.942860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:09.943148 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:09.943322 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:09.944106 (+ 784us) server_negotiation.cc:299] Negotiation successful
0811 03:08:09.944365 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":743,"threads_started":1}
I20260811 03:08:09.945902 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:09.925930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:40293)
0811 03:08:09.926962 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:09.927019 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:09.928733 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:09.929544 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:09.929552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:09.930085 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:09.930667 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:09.930750 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:09.931821 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:09.931825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:09.933077 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:09.933091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:09.933950 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:09.933959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:09.934208 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:09.935046 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:09.935074 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:09.937823 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:09.941470 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:09.941481 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:09.941501 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:09.942004 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:09.942434 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:09.942439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:09.942443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:09.942660 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:09.943295 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:09.943304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:09.943708 (+ 404us) client_negotiation.cc:769] Sending connection context
0811 03:08:09.944000 (+ 292us) client_negotiation.cc:240] Negotiation successful
0811 03:08:09.944643 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":486,"threads_started":1}
I20260811 03:08:09.947772 11729 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32863
I20260811 03:08:09.948215 11729 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:09.949061 11729 heartbeater.cc:511] Master 127.27.109.62:32863 requested a full tablet report, sending...
I20260811 03:08:09.950803 11370 ts_manager.cc:194] Registered new tserver with Master: 900519cb725642278519e3032efa2a6e (127.27.109.2:45895)
I20260811 03:08:09.951748 11370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:40293
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:08:09.995110 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32863
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:10.360379 11735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:10.360733 11735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:10.360908 11735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:10.368710 11735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:10.368889 11735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:10.369087 11735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:08:10.379920 11735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:10.381012 11583 heartbeater.cc:503] Master 127.27.109.62:32863 was elected leader, sending a full tablet report...
I20260811 03:08:10.381553 11735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:10.383281 11735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:10.397806 11740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:10.397758 11741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:10.397805 11743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:10.399029 11735 server_base.cc:1034] running on GCE node
I20260811 03:08:10.399713 11735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:10.400731 11735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:10.401953 11735 hybrid_clock.cc:648] HybridClock initialized: now 1786417690401901 us; error 37 us; skew 500 ppm
I20260811 03:08:10.402261 11735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:10 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:10.405736 11735 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:08:10.407593 11735 webserver.cc:460] Webserver started at http://127.27.109.3:40775/ using document root <none> and password file <none>
I20260811 03:08:10.408332 11735 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:10.408474 11735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:10.408834 11735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:10.412132 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a"
format_stamp: "Formatted at 2026-08-11 03:08:10 on dist-test-slave-8tdl"
server_key: "570764b973a110cc615eaa1a25368afa"
server_key_iv: "7cac46592ca33d72911e883f731c8577"
server_key_version: "encryptionkey@0"
I20260811 03:08:10.412997 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a"
format_stamp: "Formatted at 2026-08-11 03:08:10 on dist-test-slave-8tdl"
server_key: "570764b973a110cc615eaa1a25368afa"
server_key_iv: "7cac46592ca33d72911e883f731c8577"
server_key_version: "encryptionkey@0"
I20260811 03:08:10.418569 11735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:08:10.422257 11752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:10.423733 11735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:08:10.423914 11735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a"
format_stamp: "Formatted at 2026-08-11 03:08:10 on dist-test-slave-8tdl"
server_key: "570764b973a110cc615eaa1a25368afa"
server_key_iv: "7cac46592ca33d72911e883f731c8577"
server_key_version: "encryptionkey@0"
I20260811 03:08:10.424129 11735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:10.443270 11735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:10.447661 11735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:10.447970 11735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:10.449059 11735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:10.450495 11735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:10.450609 11735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:10.450783 11735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:10.450860 11735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:10.487589 11735 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:34997
I20260811 03:08:10.487685 11878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:34997 every 8 connection(s)
I20260811 03:08:10.489001 11735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:08:10.490152 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 11735
I20260811 03:08:10.490352 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:08:10.490969 28084 external_mini_cluster.cc:1428] Setting key 7d2d4e93598b3ae64b7480300f1ca0d0
Aug 11 03:08:10 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417690, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:10.512503 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.492196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:34169 (local address 127.27.109.62:32863)
0811 03:08:10.495137 (+ 2941us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.495144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.495168 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:08:10.495910 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.495917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.496023 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.496145 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:10.498169 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.499026 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.500785 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.501277 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.501380 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.505389 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:10.505394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:10.505397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:10.505434 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:10.508842 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:10.509864 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:10.509868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:10.509870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:10.510030 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:10.510610 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:10.510613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:10.510615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:10.510962 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:10.511104 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.511879 (+ 775us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.512107 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2819,"thread_start_us":2683,"threads_started":1}
I20260811 03:08:10.513588 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.492687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:34169)
0811 03:08:10.493555 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:10.493621 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:10.495550 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:10.496480 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:10.496493 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:10.497380 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:10.497994 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.498012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.499181 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.499184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.500624 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.500639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.501541 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.501548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.501786 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.502552 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:10.502586 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:10.505132 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:10.509010 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:10.509025 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:10.509041 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:10.509737 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:10.510186 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:10.510192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:10.510195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:10.510475 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:10.511080 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:10.511088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:10.511451 (+ 363us) client_negotiation.cc:769] Sending connection context
0811 03:08:10.511772 (+ 321us) client_negotiation.cc:240] Negotiation successful
0811 03:08:10.512325 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":366,"threads_started":1}
I20260811 03:08:10.515288 11879 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32863
I20260811 03:08:10.515650 11879 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:10.516521 11879 heartbeater.cc:511] Master 127.27.109.62:32863 requested a full tablet report, sending...
I20260811 03:08:10.518344 11370 ts_manager.cc:194] Registered new tserver with Master: b07eb5cb1d9f415dbad0d7a7e0a6ba5a (127.27.109.3:34997)
I20260811 03:08:10.519263 11370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:34169
I20260811 03:08:10.523080 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:08:10.542382 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.528181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.27.109.62:32863)
0811 03:08:10.528425 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.528437 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.528884 (+ 447us) server_negotiation.cc:407] Connection header received
0811 03:08:10.529063 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.529067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.529153 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.529243 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:10.530968 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.531774 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.533312 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.533729 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.533831 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.536534 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:10.536539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:10.536543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:10.536582 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:10.539571 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:10.540323 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:10.540328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:10.540330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:10.540471 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:10.540819 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:10.540823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:10.540825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:10.541207 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:10.541400 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.541770 (+ 370us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.541962 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 03:08:10.547119 11370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:08:10.554777 11370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:08:10.581840 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.569670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.27.109.2:45895)
0811 03:08:10.571371 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.571385 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.572315 (+ 930us) server_negotiation.cc:407] Connection header received
0811 03:08:10.572478 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.572491 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.572861 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.573114 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:10.574618 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.576544 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.579189 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.580346 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.580437 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.580598 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.581261 (+ 663us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.581495 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1529,"thread_start_us":1424,"threads_started":1}
I20260811 03:08:10.582861 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.569110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45895 (local address 127.0.0.1:41772)
0811 03:08:10.571946 (+ 2836us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:10.571972 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:10.572142 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:10.573870 (+ 1728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:10.573881 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:10.573917 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:10.574397 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.574410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.576748 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.576753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.578983 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.578996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.580775 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.580783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.580955 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.581000 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 03:08:10.582151 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0811 03:08:10.582375 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2655,"mutex_wait_us":173,"thread_start_us":4422,"threads_started":1}
I20260811 03:08:10.584014 11892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.568661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:43711 (local address 127.0.0.1:51692)
0811 03:08:10.569723 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:10.569778 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:10.570043 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:10.576651 (+ 6608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:10.576657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:10.576698 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:10.577285 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.577299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.579209 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.579215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.581400 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.581414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.583056 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.583065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.583278 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.583318 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 03:08:10.583388 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 03:08:10.583540 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":66,"thread_start_us":1629,"threads_started":1}
I20260811 03:08:10.584061 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.569261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:34997 (local address 127.0.0.1:53518)
0811 03:08:10.573846 (+ 4585us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:10.573863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:10.573977 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:10.576254 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:10.576258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:10.576285 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:10.576750 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.576763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.578791 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.578796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.581228 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.581245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.583171 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.583181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.583369 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.583406 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 03:08:10.583480 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 03:08:10.583612 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4446,"mutex_wait_us":7,"thread_start_us":2297,"threads_started":1}
I20260811 03:08:10.585071 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.568900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.27.109.1:43711)
0811 03:08:10.575998 (+ 7098us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.576007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.576039 (+ 32us) server_negotiation.cc:407] Connection header received
0811 03:08:10.576148 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.576158 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.576391 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.576677 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:10.577498 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.579023 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.581567 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.582760 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.584412 (+ 1652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.584649 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.584714 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.584840 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6764,"thread_start_us":6583,"threads_started":1}
I20260811 03:08:10.585263 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.570078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.27.109.3:34997)
0811 03:08:10.574649 (+ 4571us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.574659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.574736 (+ 77us) server_negotiation.cc:407] Connection header received
0811 03:08:10.574856 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.574863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.576030 (+ 1167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.576251 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:10.577005 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.578592 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.581402 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.582891 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.584419 (+ 1528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.584667 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.584737 (+ 70us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.584904 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4355,"thread_start_us":4945,"threads_started":1}
I20260811 03:08:10.587277 11658 tablet_service.cc:1467] Processing CreateTablet for tablet f64cf202d4354f129beb4baa615447fe (DEFAULT_TABLE table=test-table [id=205832fae25a4a5ba91b0b5cba1772f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:10.588764 11658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64cf202d4354f129beb4baa615447fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:10.588804 11509 tablet_service.cc:1467] Processing CreateTablet for tablet f64cf202d4354f129beb4baa615447fe (DEFAULT_TABLE table=test-table [id=205832fae25a4a5ba91b0b5cba1772f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:10.588807 11805 tablet_service.cc:1467] Processing CreateTablet for tablet f64cf202d4354f129beb4baa615447fe (DEFAULT_TABLE table=test-table [id=205832fae25a4a5ba91b0b5cba1772f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:10.590308 11509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64cf202d4354f129beb4baa615447fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:10.590409 11805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64cf202d4354f129beb4baa615447fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:10.604553 11900 tablet_bootstrap.cc:492] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Bootstrap starting.
I20260811 03:08:10.606773 11902 tablet_bootstrap.cc:492] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Bootstrap starting.
I20260811 03:08:10.607789 11901 tablet_bootstrap.cc:492] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: Bootstrap starting.
I20260811 03:08:10.608943 11900 tablet_bootstrap.cc:654] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:10.610625 11900 log.cc:824] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:10.611073 11902 tablet_bootstrap.cc:654] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:10.612197 11901 tablet_bootstrap.cc:654] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:10.612686 11902 log.cc:824] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:10.613688 11901 log.cc:824] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:10.614725 11900 tablet_bootstrap.cc:492] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: No bootstrap required, opened a new log
I20260811 03:08:10.615072 11900 ts_tablet_manager.cc:1397] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260811 03:08:10.616571 11902 tablet_bootstrap.cc:492] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: No bootstrap required, opened a new log
I20260811 03:08:10.616884 11902 ts_tablet_manager.cc:1397] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260811 03:08:10.617717 11901 tablet_bootstrap.cc:492] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: No bootstrap required, opened a new log
I20260811 03:08:10.618014 11901 ts_tablet_manager.cc:1397] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260811 03:08:10.626364 11900 raft_consensus.cc:348] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.626955 11900 raft_consensus.cc:374] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:10.627069 11900 raft_consensus.cc:729] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900519cb725642278519e3032efa2a6e, State: Initialized, Role: FOLLOWER
I20260811 03:08:10.626888 11901 raft_consensus.cc:348] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.627410 11901 raft_consensus.cc:374] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:10.627557 11901 raft_consensus.cc:729] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def07bb84f0345a1baffd3df05e63670, State: Initialized, Role: FOLLOWER
I20260811 03:08:10.627900 11900 consensus_queue.cc:260] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.628096 11902 raft_consensus.cc:348] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.628543 11902 raft_consensus.cc:374] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:10.628780 11902 raft_consensus.cc:729] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07eb5cb1d9f415dbad0d7a7e0a6ba5a, State: Initialized, Role: FOLLOWER
I20260811 03:08:10.628608 11901 consensus_queue.cc:260] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.629456 11902 consensus_queue.cc:260] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.629770 11729 heartbeater.cc:503] Master 127.27.109.62:32863 was elected leader, sending a full tablet report...
I20260811 03:08:10.630406 11900 ts_tablet_manager.cc:1428] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260811 03:08:10.631454 11879 heartbeater.cc:503] Master 127.27.109.62:32863 was elected leader, sending a full tablet report...
I20260811 03:08:10.631960 11902 ts_tablet_manager.cc:1428] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260811 03:08:10.633515 11901 ts_tablet_manager.cc:1428] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260811 03:08:10.674960 11730 tablet.cc:2369] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:10.677944 11908 raft_consensus.cc:482] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:08:10.678251 11908 raft_consensus.cc:504] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.680094 11908 leader_election.cc:288] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900519cb725642278519e3032efa2a6e (127.27.109.2:45895), b07eb5cb1d9f415dbad0d7a7e0a6ba5a (127.27.109.3:34997)
I20260811 03:08:10.690330 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.680721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50033 (local address 127.27.109.3:34997)
0811 03:08:10.680967 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.680976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.681524 (+ 548us) server_negotiation.cc:407] Connection header received
0811 03:08:10.681753 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.681759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.681840 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.681943 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:10.682908 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.684311 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.687470 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.688815 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.688992 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.689067 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.689839 (+ 772us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.689999 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 03:08:10.690330 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.680736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:38221 (local address 127.27.109.2:45895)
0811 03:08:10.681009 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.681018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.681596 (+ 578us) server_negotiation.cc:407] Connection header received
0811 03:08:10.681753 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.681759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.681840 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.681942 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:10.682908 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.684198 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.687467 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.688792 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.688987 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.689068 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.689839 (+ 771us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.689999 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 03:08:10.691087 11910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.680482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45895 (local address 127.27.109.1:38221)
0811 03:08:10.681492 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:10.681507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:10.681614 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:10.681985 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:10.681990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:10.682019 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:10.682702 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.682719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.684409 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.684415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.687238 (+ 2823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.687247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.689198 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.689204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.689519 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.689626 (+ 107us) client_negotiation.cc:769] Sending connection context
0811 03:08:10.689760 (+ 134us) client_negotiation.cc:240] Negotiation successful
0811 03:08:10.689995 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":285,"thread_start_us":542,"threads_started":1}
I20260811 03:08:10.691133 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.680422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:34997 (local address 127.27.109.1:50033)
0811 03:08:10.681388 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:10.681411 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:10.681592 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:10.681985 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:10.681990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:10.682054 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:10.682714 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.682723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.684463 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.684473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.687228 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:10.687247 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.689182 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.689190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:10.689523 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.689600 (+ 77us) client_negotiation.cc:769] Sending connection context
0811 03:08:10.689760 (+ 160us) client_negotiation.cc:240] Negotiation successful
0811 03:08:10.689995 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":319,"spinlock_wait_cycles":37760,"thread_start_us":775,"threads_started":1}
I20260811 03:08:10.692423 11827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64cf202d4354f129beb4baa615447fe" candidate_uuid: "def07bb84f0345a1baffd3df05e63670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" is_pre_election: true
I20260811 03:08:10.692423 11680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64cf202d4354f129beb4baa615447fe" candidate_uuid: "def07bb84f0345a1baffd3df05e63670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900519cb725642278519e3032efa2a6e" is_pre_election: true
I20260811 03:08:10.692989 11680 raft_consensus.cc:2454] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate def07bb84f0345a1baffd3df05e63670 in term 0.
I20260811 03:08:10.692988 11827 raft_consensus.cc:2454] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate def07bb84f0345a1baffd3df05e63670 in term 0.
I20260811 03:08:10.693734 11458 leader_election.cc:302] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 900519cb725642278519e3032efa2a6e, b07eb5cb1d9f415dbad0d7a7e0a6ba5a, def07bb84f0345a1baffd3df05e63670; no voters:
I20260811 03:08:10.694209 11908 raft_consensus.cc:2781] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:08:10.694378 11908 raft_consensus.cc:482] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:08:10.694509 11908 raft_consensus.cc:3037] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:10.696923 11908 raft_consensus.cc:504] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.697629 11908 leader_election.cc:288] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [CANDIDATE]: Term 1 election: Requested vote from peers 900519cb725642278519e3032efa2a6e (127.27.109.2:45895), b07eb5cb1d9f415dbad0d7a7e0a6ba5a (127.27.109.3:34997)
I20260811 03:08:10.698208 11680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64cf202d4354f129beb4baa615447fe" candidate_uuid: "def07bb84f0345a1baffd3df05e63670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900519cb725642278519e3032efa2a6e"
I20260811 03:08:10.698292 11827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64cf202d4354f129beb4baa615447fe" candidate_uuid: "def07bb84f0345a1baffd3df05e63670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a"
I20260811 03:08:10.698506 11827 raft_consensus.cc:3037] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:10.698484 11680 raft_consensus.cc:3037] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:10.702112 11680 raft_consensus.cc:2454] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def07bb84f0345a1baffd3df05e63670 in term 1.
I20260811 03:08:10.702112 11827 raft_consensus.cc:2454] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def07bb84f0345a1baffd3df05e63670 in term 1.
I20260811 03:08:10.702749 11458 leader_election.cc:302] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 900519cb725642278519e3032efa2a6e, b07eb5cb1d9f415dbad0d7a7e0a6ba5a, def07bb84f0345a1baffd3df05e63670; no voters:
I20260811 03:08:10.703222 11908 raft_consensus.cc:2781] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:08:10.703784 11908 raft_consensus.cc:686] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 1 LEADER]: Becoming Leader. State: Replica: def07bb84f0345a1baffd3df05e63670, State: Running, Role: LEADER
I20260811 03:08:10.704305 11908 consensus_queue.cc:237] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:10.709357 11367 catalog_manager.cc:5495] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 reported cstate change: term changed from 0 to 1, leader changed from <none> to def07bb84f0345a1baffd3df05e63670 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "def07bb84f0345a1baffd3df05e63670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } health_report { overall_health: UNKNOWN } } }
W20260811 03:08:10.740861 11880 tablet.cc:2369] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:10.743597 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.737830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.27.109.1:43711)
0811 03:08:10.738048 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.738053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.738090 (+ 37us) server_negotiation.cc:407] Connection header received
0811 03:08:10.738188 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.738192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.738273 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.738401 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:08:10.738880 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.739646 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.741216 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.741670 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.741821 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.742381 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:08:10.742979 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:08:10.743158 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.743221 (+ 63us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.743312 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 03:08:10.752983 11827 raft_consensus.cc:1261] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 1 FOLLOWER]: Refusing update from remote peer def07bb84f0345a1baffd3df05e63670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:10.753180 11680 raft_consensus.cc:1261] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 1 FOLLOWER]: Refusing update from remote peer def07bb84f0345a1baffd3df05e63670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:10.754355 11912 consensus_queue.cc:1035] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:10.754782 11908 consensus_queue.cc:1035] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:10.769383 11915 mvcc.cc:205] Tried to move back new op lower bound from 7317166861312405504 to 7317166861128699904. Current Snapshot: MvccSnapshot[applied={T|T < 7317166861312405504}]
I20260811 03:08:10.787451 11370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:08:10.787905 11370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:08:10.792591 11370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Sending DeleteTablet for 3 replicas of tablet f64cf202d4354f129beb4baa615447fe
I20260811 03:08:10.793943 11509 tablet_service.cc:1514] Processing DeleteTablet for tablet f64cf202d4354f129beb4baa615447fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:10 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:51692
I20260811 03:08:10.794008 11658 tablet_service.cc:1514] Processing DeleteTablet for tablet f64cf202d4354f129beb4baa615447fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:10 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:41772
I20260811 03:08:10.794875 11805 tablet_service.cc:1514] Processing DeleteTablet for tablet f64cf202d4354f129beb4baa615447fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:10 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:53518
I20260811 03:08:10.795138 11925 tablet_replica.cc:330] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: stopping tablet replica
I20260811 03:08:10.795799 11925 raft_consensus.cc:2229] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:08:10.796813 11925 raft_consensus.cc:2258] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:10.799155 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 11441
I20260811 03:08:10.800009 11925 ts_tablet_manager.cc:1905] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:10.811789 11929 tablet_replica.cc:330] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: stopping tablet replica
I20260811 03:08:10.812582 11929 raft_consensus.cc:2229] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:08:10.813299 11929 raft_consensus.cc:2258] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:10.814440 11927 tablet_replica.cc:330] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: stopping tablet replica
I20260811 03:08:10.815030 11927 raft_consensus.cc:2229] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:08:10.815694 11927 raft_consensus.cc:2258] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:10.816398 11929 ts_tablet_manager.cc:1905] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:10.818588 11925 ts_tablet_manager.cc:1918] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:10.818837 11925 log.cc:1196] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/f64cf202d4354f129beb4baa615447fe
I20260811 03:08:10.819551 11925 ts_tablet_manager.cc:1939] T f64cf202d4354f129beb4baa615447fe P def07bb84f0345a1baffd3df05e63670: Deleting consensus metadata
I20260811 03:08:10.820662 11927 ts_tablet_manager.cc:1905] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:10.821877 11355 catalog_manager.cc:4841] TS def07bb84f0345a1baffd3df05e63670 (127.27.109.1:43711): tablet f64cf202d4354f129beb4baa615447fe (table test-table [id=205832fae25a4a5ba91b0b5cba1772f1]) successfully deleted
Aug 11 03:08:10 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:08:10.825320 11929 ts_tablet_manager.cc:1918] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:10.825552 11929 log.cc:1196] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/f64cf202d4354f129beb4baa615447fe
I20260811 03:08:10.826233 11929 ts_tablet_manager.cc:1939] T f64cf202d4354f129beb4baa615447fe P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Deleting consensus metadata
I20260811 03:08:10.828737 11351 catalog_manager.cc:4841] TS b07eb5cb1d9f415dbad0d7a7e0a6ba5a (127.27.109.3:34997): tablet f64cf202d4354f129beb4baa615447fe (table test-table [id=205832fae25a4a5ba91b0b5cba1772f1]) successfully deleted
I20260811 03:08:10.831985 11927 ts_tablet_manager.cc:1918] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:10.832196 11927 log.cc:1196] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/f64cf202d4354f129beb4baa615447fe
I20260811 03:08:10.832895 11927 ts_tablet_manager.cc:1939] T f64cf202d4354f129beb4baa615447fe P 900519cb725642278519e3032efa2a6e: Deleting consensus metadata
I20260811 03:08:10.835106 11354 catalog_manager.cc:4841] TS 900519cb725642278519e3032efa2a6e (127.27.109.2:45895): tablet f64cf202d4354f129beb4baa615447fe (table test-table [id=205832fae25a4a5ba91b0b5cba1772f1]) successfully deleted
I20260811 03:08:10.835312 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.800805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.27.109.1:43711)
0811 03:08:10.801028 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.801041 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.808261 (+ 7220us) server_negotiation.cc:407] Connection header received
0811 03:08:10.808767 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.808786 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.808913 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.809024 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:10.811951 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.813181 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.816487 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.817227 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.817338 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.826404 (+ 9066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:10.826425 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:10.826446 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:10.826507 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:10.831156 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:10.832361 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:10.832374 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:10.832379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:10.832629 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:10.833204 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:10.833213 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:10.833219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:10.833753 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:10.834101 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.834496 (+ 395us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.834836 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 03:08:10.836196 11578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:10.942169 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 11441
I20260811 03:08:10.966877 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 11590
Aug 11 03:08:10 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:08:10.983889 11897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:10.967795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.27.109.2:45895)
0811 03:08:10.967973 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:10.967978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:10.968018 (+ 40us) server_negotiation.cc:407] Connection header received
0811 03:08:10.968088 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:10.968092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:10.968156 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:10.968245 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:10.969725 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.970781 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.972224 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:10.972651 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:10.972765 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:10.977388 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:10.977407 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:10.977424 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:10.977467 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:10.980729 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:10.981614 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:10.981623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:10.981625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:10.981762 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:10.982193 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:10.982197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:10.982199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:10.982626 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:10.982989 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:10.983304 (+ 315us) server_negotiation.cc:299] Negotiation successful
0811 03:08:10.983485 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 03:08:10.984798 11724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:11.082093 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 11590
I20260811 03:08:11.104393 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 11735
Aug 11 03:08:11 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:08:11.120880 11898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:11.105260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.27.109.3:34997)
0811 03:08:11.105456 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:11.105461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:11.105480 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:08:11.105588 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:11.105592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:11.105670 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:11.105756 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:11.107365 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:11.108270 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:11.109684 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:11.110076 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:11.110172 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:11.115188 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:11.115207 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:11.115225 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:11.115266 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:11.118127 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:11.118925 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:11.118932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:11.118935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:11.119125 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:11.119478 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:11.119481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:11.119483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:11.119873 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:11.120134 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:11.120428 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 03:08:11.120627 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 03:08:11.121555 11873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:11.217504 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 11735
I20260811 03:08:11.240010 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 11331
I20260811 03:08:11.241153 11406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:11.359056 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 11331
I20260811 03:08:11.388465 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32863
--webserver_interface=127.27.109.62
--webserver_port=43105
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:32863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:11.761440 11936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:11.761785 11936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:11.761880 11936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:11.769657 11936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:11.769786 11936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:11.769835 11936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:11.769888 11936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:11.769935 11936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:11.780884 11936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:32863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=43105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:11.782333 11936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:11.783722 11936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:11.816166 11943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:11.816296 11945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:11.816319 11942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:11.817189 11936 server_base.cc:1034] running on GCE node
I20260811 03:08:11.818213 11936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:11.820269 11936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:11.821537 11936 hybrid_clock.cc:648] HybridClock initialized: now 1786417691821476 us; error 58 us; skew 500 ppm
I20260811 03:08:11.822044 11936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:11 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417691, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:11.831432 11936 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:08:11.833379 11936 webserver.cc:460] Webserver started at http://127.27.109.62:43105/ using document root <none> and password file <none>
I20260811 03:08:11.834245 11936 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:11.834421 11936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:11.841610 11936 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260811 03:08:11.845196 11953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:11.846853 11936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 03:08:11.847069 11936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "836425ae2dee4a99b606933df3f7e0a6"
format_stamp: "Formatted at 2026-08-11 03:08:08 on dist-test-slave-8tdl"
server_key: "2460ed5a69e453d623a4f6c1421fe905"
server_key_iv: "100b93d8b89b9dd925c82d42fbc8f234"
server_key_version: "encryptionkey@0"
I20260811 03:08:11.848321 11936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:11.865411 11936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:11.870425 11936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:11.870797 11936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:11.892351 11936 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:32863
I20260811 03:08:11.892377 12006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:32863 every 8 connection(s)
I20260811 03:08:11.894347 11936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:08:11.902479 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 11936
I20260811 03:08:11.902510 12007 sys_catalog.cc:263] Verifying existing consensus state
I20260811 03:08:11.903600 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:43711
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:32863
--webserver_port=44451
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:08:11.908332 12007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Bootstrap starting.
I20260811 03:08:11.932744 12007 log.cc:824] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:11.946590 12007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 03:08:11.947227 12007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Bootstrap complete.
I20260811 03:08:11.957671 12007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:11.958663 12007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836425ae2dee4a99b606933df3f7e0a6, State: Initialized, Role: FOLLOWER
I20260811 03:08:11.959475 12007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:11.959769 12007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:08:11.959924 12007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:08:11.960132 12007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 1 FOLLOWER]: Advancing to term 2
I20260811 03:08:11.964557 12007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:11.965165 12007 leader_election.cc:302] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 836425ae2dee4a99b606933df3f7e0a6; no voters:
I20260811 03:08:11.965898 12007 leader_election.cc:288] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 03:08:11.966107 12012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 03:08:11.967402 12012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 2 LEADER]: Becoming Leader. State: Replica: 836425ae2dee4a99b606933df3f7e0a6, State: Running, Role: LEADER
I20260811 03:08:11.968006 12012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:11.968425 12007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:08:11.971021 12013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "836425ae2dee4a99b606933df3f7e0a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } } }
I20260811 03:08:11.971560 12013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:11.976193 12014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836425ae2dee4a99b606933df3f7e0a6. Latest consensus state: current_term: 2 leader_uuid: "836425ae2dee4a99b606933df3f7e0a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } } }
I20260811 03:08:11.976577 12014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:11.979658 12017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:08:11.989665 12017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:08:11.991497 12017 catalog_manager.cc:1259] Loaded cluster ID: 9b130beb2c3a402da9b24be0af2828cd
I20260811 03:08:11.991706 12017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:08:11.996273 12017 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260811 03:08:12.017798 12030 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 03:08:12.023329 12017 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260811 03:08:12.031944 12017 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:08:12.033727 12017 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Loaded TSK: 0
I20260811 03:08:12.035105 12017 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 03:08:12.328583 12009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:12.328958 12009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:12.329131 12009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:12.336937 12009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:12.337122 12009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:12.337311 12009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:08:12.348029 12009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:43711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=44451
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:12.349502 12009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:12.351059 12009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:12.366887 12038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:12.367051 12037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:12.367408 12040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:12.369510 12009 server_base.cc:1034] running on GCE node
I20260811 03:08:12.370282 12009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:12.371592 12009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:12.372843 12009 hybrid_clock.cc:648] HybridClock initialized: now 1786417692372783 us; error 54 us; skew 500 ppm
I20260811 03:08:12.373292 12009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:12 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417692, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:12.376879 12009 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:08:12.378799 12009 webserver.cc:460] Webserver started at http://127.27.109.1:44451/ using document root <none> and password file <none>
I20260811 03:08:12.379631 12009 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:12.379797 12009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:12.386159 12009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260811 03:08:12.389514 12047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:12.391364 12009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260811 03:08:12.391577 12009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "def07bb84f0345a1baffd3df05e63670"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "39b20b899cd49795c10d4498a423f1a1"
server_key_iv: "1eacb5ea17329b23f83d55711a61e92c"
server_key_version: "encryptionkey@0"
I20260811 03:08:12.392693 12009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:12.410977 12009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:12.415890 12009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:12.416217 12009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:12.417414 12009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:12.418965 12009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:12.419065 12009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:12.419204 12009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:12.419310 12009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:12.454723 12009 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:43711
I20260811 03:08:12.454816 12168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:43711 every 8 connection(s)
I20260811 03:08:12.456301 12009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:08:12.463164 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 12009
I20260811 03:08:12.464113 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:45895
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:32863
--webserver_port=38303
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:08:12 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417692, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:12.485311 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:12.459480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:38365)
0811 03:08:12.460351 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:12.460406 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:12.462235 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:12.464109 (+ 1874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:12.464117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:12.464780 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:12.465378 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:12.465396 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:12.467910 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:12.467914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:12.469324 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:12.469337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:12.470951 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:12.470956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:12.471187 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:12.471940 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:12.471972 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:12.474748 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:12.480080 (+ 5332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:12.480090 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:12.480104 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:12.480801 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:12.481450 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:12.481473 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:12.481480 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:12.481727 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:12.482877 (+ 1150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:12.482884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:12.483259 (+ 375us) client_negotiation.cc:769] Sending connection context
0811 03:08:12.483544 (+ 285us) client_negotiation.cc:240] Negotiation successful
0811 03:08:12.484043 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":364,"threads_started":1}
I20260811 03:08:12.486294 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:12.459345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:38365 (local address 127.27.109.62:32863)
0811 03:08:12.459998 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:12.460011 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:12.460555 (+ 544us) server_negotiation.cc:407] Connection header received
0811 03:08:12.462932 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:12.462954 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:12.463473 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:12.463956 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:12.465652 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:12.467732 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:12.469489 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:12.470698 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:12.470927 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:12.475109 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:12.475120 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:12.475138 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:12.475183 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:12.479693 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:12.480986 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:12.480999 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:12.481010 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:12.481270 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:12.481864 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:12.481872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:12.481878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:12.482715 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:12.482922 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:12.483826 (+ 904us) server_negotiation.cc:299] Negotiation successful
0811 03:08:12.484345 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":317,"threads_started":1}
I20260811 03:08:12.489693 12169 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32863
I20260811 03:08:12.490154 12169 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:12.491077 12169 heartbeater.cc:511] Master 127.27.109.62:32863 requested a full tablet report, sending...
I20260811 03:08:12.494087 11970 ts_manager.cc:194] Registered new tserver with Master: def07bb84f0345a1baffd3df05e63670 (127.27.109.1:43711)
I20260811 03:08:12.496644 11970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:38365
W20260811 03:08:12.835035 12174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:12.835348 12174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:12.835448 12174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:12.843150 12174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:12.843271 12174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:12.843387 12174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:08:12.853753 12174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:45895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=38303
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:12.855278 12174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:12.856742 12174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:12.871376 12182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:12.871357 12181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:12.872253 12184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:12.873209 12174 server_base.cc:1034] running on GCE node
I20260811 03:08:12.873939 12174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:12.875103 12174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:12.876361 12174 hybrid_clock.cc:648] HybridClock initialized: now 1786417692876296 us; error 60 us; skew 500 ppm
I20260811 03:08:12.876724 12174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:12 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417692, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:12.880137 12174 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:08:12.881826 12174 webserver.cc:460] Webserver started at http://127.27.109.2:38303/ using document root <none> and password file <none>
I20260811 03:08:12.882552 12174 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:12.882722 12174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:12.888738 12174 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260811 03:08:12.892230 12191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:12.893772 12174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 03:08:12.893982 12174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "900519cb725642278519e3032efa2a6e"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "2933851f6e3be2a826c85d05ae441376"
server_key_iv: "2d60a2c94eb760022cb5d5ff243e4da5"
server_key_version: "encryptionkey@0"
I20260811 03:08:12.895166 12174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:12.909150 12174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:12.913755 12174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:12.914057 12174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:12.915160 12174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:12.916556 12174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:12.916683 12174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:12.916815 12174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:12.916904 12174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:12.953662 12174 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45895
I20260811 03:08:12.953718 12319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45895 every 8 connection(s)
I20260811 03:08:12.955346 12174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:08:12.962066 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 12174
I20260811 03:08:12.962884 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:34997
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:32863
--webserver_port=40775
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:08:12 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417692, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:12.980295 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:12.958199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:51979 (local address 127.27.109.62:32863)
0811 03:08:12.958382 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:12.958387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:12.959549 (+ 1162us) server_negotiation.cc:407] Connection header received
0811 03:08:12.961644 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:12.961648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:12.961736 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:12.961830 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:12.964066 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:12.964828 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:12.967580 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:12.968029 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:12.968209 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:12.973031 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:12.973037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:12.973041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:12.973084 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:12.976644 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:12.977451 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:12.977456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:12.977458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:12.977636 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:12.978145 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:12.978149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:12.978151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:12.978437 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:12.978609 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:12.979514 (+ 905us) server_negotiation.cc:299] Negotiation successful
0811 03:08:12.979803 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 03:08:12.981148 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:12.958555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:51979)
0811 03:08:12.959357 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:12.959410 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:12.961338 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:12.962142 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:12.962156 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:12.962968 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:12.963838 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:12.963868 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:12.965022 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:12.965030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:12.967394 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:12.967412 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:12.968276 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:12.968286 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:12.968630 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:12.969493 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:12.969529 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:12.972745 (+ 3216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:12.976813 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:12.976820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:12.976834 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:12.977294 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:12.977758 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:12.977763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:12.977766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:12.977963 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:12.978579 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:12.978587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:12.979018 (+ 431us) client_negotiation.cc:769] Sending connection context
0811 03:08:12.979344 (+ 326us) client_negotiation.cc:240] Negotiation successful
0811 03:08:12.979820 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":330,"threads_started":1}
I20260811 03:08:12.982729 12320 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32863
I20260811 03:08:12.983070 12320 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:12.983861 12320 heartbeater.cc:511] Master 127.27.109.62:32863 requested a full tablet report, sending...
I20260811 03:08:12.985499 11970 ts_manager.cc:194] Registered new tserver with Master: 900519cb725642278519e3032efa2a6e (127.27.109.2:45895)
I20260811 03:08:12.986438 11970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:51979
W20260811 03:08:13.329725 12323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:13.330120 12323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:13.330314 12323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:13.338016 12323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:13.338213 12323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:13.338397 12323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:08:13.348737 12323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:34997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=40775
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:13.350279 12323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:13.351755 12323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:13.366771 12329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:13.366667 12330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:13.366722 12332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:13.368288 12323 server_base.cc:1034] running on GCE node
I20260811 03:08:13.368970 12323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:13.370118 12323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:13.371356 12323 hybrid_clock.cc:648] HybridClock initialized: now 1786417693371284 us; error 63 us; skew 500 ppm
I20260811 03:08:13.371706 12323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:13 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:13.378079 12323 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:08:13.380373 12323 webserver.cc:460] Webserver started at http://127.27.109.3:40775/ using document root <none> and password file <none>
I20260811 03:08:13.381476 12323 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:13.381654 12323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:13.390513 12323 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 03:08:13.395051 12339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:13.397197 12323 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:08:13.397423 12323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a"
format_stamp: "Formatted at 2026-08-11 03:08:10 on dist-test-slave-8tdl"
server_key: "570764b973a110cc615eaa1a25368afa"
server_key_iv: "7cac46592ca33d72911e883f731c8577"
server_key_version: "encryptionkey@0"
I20260811 03:08:13.399216 12323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:13.418066 12323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:13.424026 12323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:13.424422 12323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:13.425794 12323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:13.427322 12323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:13.427425 12323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:13.427573 12323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:13.427652 12323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:13.465076 12323 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:34997
I20260811 03:08:13.465169 12462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:34997 every 8 connection(s)
I20260811 03:08:13.466605 12323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:08:13.468806 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 12323
Aug 11 03:08:13 dist-test-slave-8tdl krb5kdc[11316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417693, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:13.486264 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.469698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.27.109.62:32863)
0811 03:08:13.470198 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.470206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.470669 (+ 463us) server_negotiation.cc:407] Connection header received
0811 03:08:13.470916 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.470921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.471016 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.471141 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:13.473298 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.474072 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.475602 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.476103 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.476247 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.479178 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:13.479192 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:13.479196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:13.479230 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:13.482861 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:13.483928 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:13.483934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:13.483939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:13.484129 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:13.484607 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:13.484611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:13.484613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:13.485023 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:13.485257 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.485670 (+ 413us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.485865 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
I20260811 03:08:13.490119 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.469330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:51717 (local address 127.27.109.62:32863)
0811 03:08:13.469515 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.469520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.471123 (+ 1603us) server_negotiation.cc:407] Connection header received
0811 03:08:13.473808 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.473813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.473900 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.474021 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:13.475670 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.476479 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.478083 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.478556 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.478724 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.483113 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:13.483122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:13.483128 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:13.483175 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:13.486644 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:13.487589 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:13.487594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:13.487598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:13.487778 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:13.488388 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:13.488391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:13.488393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:13.488679 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:13.488850 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.489679 (+ 829us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.489857 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 03:08:13.491253 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.470020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:51717)
0811 03:08:13.470968 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:13.471013 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:13.473539 (+ 2526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:13.474276 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:13.474284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:13.474892 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:13.475493 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.475511 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.476660 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.476665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.477922 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.477935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.478850 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.478858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.479116 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.479881 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:13.479921 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:13.482780 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:13.486847 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:13.486854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:13.486869 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:13.487440 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:13.487931 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:13.487936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:13.487941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:13.488259 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:13.488876 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:13.488885 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:13.489303 (+ 418us) client_negotiation.cc:769] Sending connection context
0811 03:08:13.489558 (+ 255us) client_negotiation.cc:240] Negotiation successful
0811 03:08:13.489983 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":346,"threads_started":1}
I20260811 03:08:13.492713 12463 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32863
I20260811 03:08:13.493046 12463 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:13.493793 12463 heartbeater.cc:511] Master 127.27.109.62:32863 requested a full tablet report, sending...
I20260811 03:08:13.495334 11970 ts_manager.cc:194] Registered new tserver with Master: b07eb5cb1d9f415dbad0d7a7e0a6ba5a (127.27.109.3:34997)
I20260811 03:08:13.496217 11970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:51717
I20260811 03:08:13.500564 12169 heartbeater.cc:503] Master 127.27.109.62:32863 was elected leader, sending a full tablet report...
I20260811 03:08:13.501703 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:08:13.508842 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.502811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.27.109.62:32863)
0811 03:08:13.502985 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.502990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.503516 (+ 526us) server_negotiation.cc:407] Connection header received
0811 03:08:13.503694 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.503698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.503827 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.503948 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:08:13.504418 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.505195 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.506787 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.507194 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.507291 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.507817 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:08:13.508345 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:08:13.508454 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.508577 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.508648 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 03:08:13.509791 11969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:08:13.517611 11969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:08:13.554114 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.530563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:43711 (local address 127.0.0.1:51712)
0811 03:08:13.542839 (+ 12276us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:13.542893 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:13.543125 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:13.545963 (+ 2838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:13.545975 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:13.546046 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:13.546820 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.546837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.548981 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.548989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.551299 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.551314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.552995 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.553002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.553197 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.553278 (+ 81us) client_negotiation.cc:769] Sending connection context
0811 03:08:13.553416 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 03:08:13.553694 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11968,"mutex_wait_us":1,"thread_start_us":11768,"threads_started":1}
I20260811 03:08:13.554584 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.531392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.27.109.1:43711)
0811 03:08:13.543969 (+ 12577us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.543982 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.544024 (+ 42us) server_negotiation.cc:407] Connection header received
0811 03:08:13.544183 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.544196 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.544538 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.544799 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:13.547111 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.548846 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.551478 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.552745 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.552918 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.553193 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.553538 (+ 345us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.553714 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12341,"thread_start_us":12153,"threads_started":1}
I20260811 03:08:13.556617 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.531997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.27.109.3:34997)
0811 03:08:13.545737 (+ 13740us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.545749 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.547088 (+ 1339us) server_negotiation.cc:407] Connection header received
0811 03:08:13.547384 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.547391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.547624 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.547819 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:13.548794 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.550384 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.553200 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.555016 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.555140 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.555379 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.556047 (+ 668us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.556268 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13514,"thread_start_us":13296,"threads_started":1}
I20260811 03:08:13.556835 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.531037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.27.109.2:45895)
0811 03:08:13.542122 (+ 11085us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.542137 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.546564 (+ 4427us) server_negotiation.cc:407] Connection header received
0811 03:08:13.547017 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.547028 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.547311 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.547481 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:13.548396 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.549984 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.553200 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.555322 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.555447 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.555666 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.556281 (+ 615us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.556471 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10799,"thread_start_us":14531,"threads_started":1}
I20260811 03:08:13.557497 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.530863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:34997 (local address 127.0.0.1:53528)
0811 03:08:13.546977 (+ 16114us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:13.546995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:13.547192 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:13.548111 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:13.548115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:13.548141 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:13.548583 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.548593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.550563 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.550569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.552989 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.553005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.555586 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.555593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.555813 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.555857 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 03:08:13.556905 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0811 03:08:13.557062 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15944,"mutex_wait_us":82,"thread_start_us":15634,"threads_started":1}
I20260811 03:08:13.557622 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.530592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45895 (local address 127.0.0.1:41788)
0811 03:08:13.546399 (+ 15807us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:13.546433 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:13.546771 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:13.547792 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:13.547797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:13.547822 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:13.548223 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.548232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.550157 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.550162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.552990 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.553005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.555878 (+ 2873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.555885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.556080 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.556111 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 03:08:13.557088 (+ 977us) client_negotiation.cc:240] Negotiation successful
0811 03:08:13.557231 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15644,"mutex_wait_us":396,"thread_start_us":15310,"threads_started":1}
I20260811 03:08:13.558557 12098 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98cf87952b4160ae15a03ed18638cc (DEFAULT_TABLE table=test-table [id=8f4fbff6fc1a47fcb1e59136e33c1fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:13.560299 12098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98cf87952b4160ae15a03ed18638cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:13.561268 12244 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98cf87952b4160ae15a03ed18638cc (DEFAULT_TABLE table=test-table [id=8f4fbff6fc1a47fcb1e59136e33c1fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:13.561369 12392 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98cf87952b4160ae15a03ed18638cc (DEFAULT_TABLE table=test-table [id=8f4fbff6fc1a47fcb1e59136e33c1fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:13.562568 12244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98cf87952b4160ae15a03ed18638cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:13.562774 12392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98cf87952b4160ae15a03ed18638cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:13.576220 12480 tablet_bootstrap.cc:492] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Bootstrap starting.
I20260811 03:08:13.578423 12481 tablet_bootstrap.cc:492] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Bootstrap starting.
I20260811 03:08:13.581504 12480 tablet_bootstrap.cc:654] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:13.582237 12481 tablet_bootstrap.cc:654] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:13.582988 12480 log.cc:824] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:13.583547 12481 log.cc:824] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:13.583611 12482 tablet_bootstrap.cc:492] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Bootstrap starting.
I20260811 03:08:13.586658 12482 tablet_bootstrap.cc:654] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:13.588200 12482 log.cc:824] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:13.590394 12480 tablet_bootstrap.cc:492] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: No bootstrap required, opened a new log
I20260811 03:08:13.590581 12481 tablet_bootstrap.cc:492] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: No bootstrap required, opened a new log
I20260811 03:08:13.590749 12480 ts_tablet_manager.cc:1397] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260811 03:08:13.590885 12481 ts_tablet_manager.cc:1397] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260811 03:08:13.591264 12482 tablet_bootstrap.cc:492] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: No bootstrap required, opened a new log
I20260811 03:08:13.591529 12482 ts_tablet_manager.cc:1397] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260811 03:08:13.599056 12480 raft_consensus.cc:348] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.599489 12480 raft_consensus.cc:374] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:13.599591 12480 raft_consensus.cc:729] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900519cb725642278519e3032efa2a6e, State: Initialized, Role: FOLLOWER
I20260811 03:08:13.600371 12480 consensus_queue.cc:260] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.601572 12320 heartbeater.cc:503] Master 127.27.109.62:32863 was elected leader, sending a full tablet report...
I20260811 03:08:13.602365 12480 ts_tablet_manager.cc:1428] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260811 03:08:13.602283 12481 raft_consensus.cc:348] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.602298 12482 raft_consensus.cc:348] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.602790 12481 raft_consensus.cc:374] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:13.602789 12482 raft_consensus.cc:374] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:13.602913 12481 raft_consensus.cc:729] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def07bb84f0345a1baffd3df05e63670, State: Initialized, Role: FOLLOWER
I20260811 03:08:13.602900 12482 raft_consensus.cc:729] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07eb5cb1d9f415dbad0d7a7e0a6ba5a, State: Initialized, Role: FOLLOWER
I20260811 03:08:13.603624 12481 consensus_queue.cc:260] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.603624 12482 consensus_queue.cc:260] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.604923 12463 heartbeater.cc:503] Master 127.27.109.62:32863 was elected leader, sending a full tablet report...
I20260811 03:08:13.606244 12482 ts_tablet_manager.cc:1428] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260811 03:08:13.609361 12481 ts_tablet_manager.cc:1428] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260811 03:08:13.707525 12321 tablet.cc:2369] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:08:13.708914 12170 tablet.cc:2369] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:08:13.718322 12464 tablet.cc:2369] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:13.836575 12489 raft_consensus.cc:482] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:08:13.837013 12489 raft_consensus.cc:504] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.838932 12489 leader_election.cc:288] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900519cb725642278519e3032efa2a6e (127.27.109.2:45895), b07eb5cb1d9f415dbad0d7a7e0a6ba5a (127.27.109.3:34997)
I20260811 03:08:13.846202 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.839517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:54613 (local address 127.27.109.3:34997)
0811 03:08:13.839773 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.839781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.840573 (+ 792us) server_negotiation.cc:407] Connection header received
0811 03:08:13.840799 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.840803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.840865 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.840961 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:13.841632 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.842385 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.844403 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.845213 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.845409 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.845503 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.845788 (+ 285us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.845920 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 03:08:13.846378 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.839225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:34997 (local address 127.27.109.1:54613)
0811 03:08:13.840432 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:13.840451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:13.840645 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:13.841009 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:13.841017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:13.841041 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:13.841469 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.841482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.842523 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.842531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.844232 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.844245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.845425 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.845431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.845619 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.845647 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 03:08:13.845755 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 03:08:13.845895 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":1,"thread_start_us":902,"threads_started":1}
I20260811 03:08:13.846398 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.839475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:44635 (local address 127.27.109.2:45895)
0811 03:08:13.839719 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.839729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.840623 (+ 894us) server_negotiation.cc:407] Connection header received
0811 03:08:13.840830 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.840835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.840900 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.841004 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:13.841697 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.842406 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.844410 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.845363 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.845552 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.845640 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.845984 (+ 344us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.846118 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260811 03:08:13.846522 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.839434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45895 (local address 127.27.109.1:44635)
0811 03:08:13.840491 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:13.840506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:13.840673 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:13.841021 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:13.841026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:13.841148 (+ 122us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:13.841529 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.841538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.842545 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.842551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.844237 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:13.844250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.845649 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.845658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:13.845792 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.845819 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 03:08:13.845951 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 03:08:13.846083 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":2,"spinlock_wait_cycles":76672,"thread_start_us":479,"threads_started":1}
I20260811 03:08:13.847672 12268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cf87952b4160ae15a03ed18638cc" candidate_uuid: "def07bb84f0345a1baffd3df05e63670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900519cb725642278519e3032efa2a6e" is_pre_election: true
I20260811 03:08:13.847672 12417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cf87952b4160ae15a03ed18638cc" candidate_uuid: "def07bb84f0345a1baffd3df05e63670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" is_pre_election: true
I20260811 03:08:13.848299 12417 raft_consensus.cc:2454] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate def07bb84f0345a1baffd3df05e63670 in term 0.
I20260811 03:08:13.848299 12268 raft_consensus.cc:2454] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate def07bb84f0345a1baffd3df05e63670 in term 0.
I20260811 03:08:13.849128 12050 leader_election.cc:302] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 900519cb725642278519e3032efa2a6e, b07eb5cb1d9f415dbad0d7a7e0a6ba5a, def07bb84f0345a1baffd3df05e63670; no voters:
I20260811 03:08:13.849623 12489 raft_consensus.cc:2781] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:08:13.849824 12489 raft_consensus.cc:482] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:08:13.849946 12489 raft_consensus.cc:3037] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:13.852550 12489 raft_consensus.cc:504] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.853266 12489 leader_election.cc:288] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [CANDIDATE]: Term 1 election: Requested vote from peers 900519cb725642278519e3032efa2a6e (127.27.109.2:45895), b07eb5cb1d9f415dbad0d7a7e0a6ba5a (127.27.109.3:34997)
I20260811 03:08:13.853943 12417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cf87952b4160ae15a03ed18638cc" candidate_uuid: "def07bb84f0345a1baffd3df05e63670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a"
I20260811 03:08:13.853842 12268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98cf87952b4160ae15a03ed18638cc" candidate_uuid: "def07bb84f0345a1baffd3df05e63670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900519cb725642278519e3032efa2a6e"
I20260811 03:08:13.854178 12417 raft_consensus.cc:3037] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:13.854178 12268 raft_consensus.cc:3037] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:13.857671 12268 raft_consensus.cc:2454] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def07bb84f0345a1baffd3df05e63670 in term 1.
I20260811 03:08:13.857671 12417 raft_consensus.cc:2454] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate def07bb84f0345a1baffd3df05e63670 in term 1.
I20260811 03:08:13.858206 12048 leader_election.cc:302] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b07eb5cb1d9f415dbad0d7a7e0a6ba5a, def07bb84f0345a1baffd3df05e63670; no voters:
I20260811 03:08:13.858603 12489 raft_consensus.cc:2781] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:08:13.859135 12489 raft_consensus.cc:686] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 1 LEADER]: Becoming Leader. State: Replica: def07bb84f0345a1baffd3df05e63670, State: Running, Role: LEADER
I20260811 03:08:13.859827 12489 consensus_queue.cc:237] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } }
I20260811 03:08:13.865504 11969 catalog_manager.cc:5495] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 reported cstate change: term changed from 0 to 1, leader changed from <none> to def07bb84f0345a1baffd3df05e63670 (127.27.109.1). New cstate: current_term: 1 leader_uuid: "def07bb84f0345a1baffd3df05e63670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "def07bb84f0345a1baffd3df05e63670" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 43711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 } health_report { overall_health: UNKNOWN } } }
I20260811 03:08:13.938849 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:13.932949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.27.109.1:43711)
0811 03:08:13.933196 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:13.933204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:13.933228 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:08:13.933520 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:13.933524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:13.933608 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:13.933774 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:08:13.934212 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.935008 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.936506 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:13.936932 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:13.937080 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:13.937611 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:08:13.938134 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:08:13.938321 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:13.938414 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 03:08:13.938510 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 03:08:13.948303 12417 raft_consensus.cc:1261] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 1 FOLLOWER]: Refusing update from remote peer def07bb84f0345a1baffd3df05e63670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:13.948451 12268 raft_consensus.cc:1261] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 1 FOLLOWER]: Refusing update from remote peer def07bb84f0345a1baffd3df05e63670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:13.949663 12489 consensus_queue.cc:1035] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:13.950148 12493 consensus_queue.cc:1035] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900519cb725642278519e3032efa2a6e" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:13.967136 12495 mvcc.cc:205] Tried to move back new op lower bound from 7317166874400112640 to 7317166874054287360. Current Snapshot: MvccSnapshot[applied={T|T < 7317166874400112640}]
I20260811 03:08:13.992198 12496 mvcc.cc:205] Tried to move back new op lower bound from 7317166874400112640 to 7317166874054287360. Current Snapshot: MvccSnapshot[applied={T|T < 7317166874400112640}]
I20260811 03:08:13.996881 11969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:08:13.997380 11969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:08:14.003298 11969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Sending DeleteTablet for 3 replicas of tablet 1c98cf87952b4160ae15a03ed18638cc
I20260811 03:08:14.006776 12098 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c98cf87952b4160ae15a03ed18638cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:13 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:51712
I20260811 03:08:14.006929 12392 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c98cf87952b4160ae15a03ed18638cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:13 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:53528
I20260811 03:08:14.007016 12243 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c98cf87952b4160ae15a03ed18638cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:13 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:41788
I20260811 03:08:14.008260 12244 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c98cf87952b4160ae15a03ed18638cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:13 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:41788
I20260811 03:08:14.012024 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 12009
I20260811 03:08:14.031358 12507 tablet_replica.cc:330] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: stopping tablet replica
I20260811 03:08:14.032114 12507 raft_consensus.cc:2229] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:08:14.032874 12507 raft_consensus.cc:2258] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:14.035161 12506 tablet_replica.cc:330] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: stopping tablet replica
I20260811 03:08:14.035795 12506 raft_consensus.cc:2229] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:08:14.036450 12506 raft_consensus.cc:2258] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:14.040493 12505 tablet_replica.cc:330] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: stopping tablet replica
I20260811 03:08:14.040995 12505 raft_consensus.cc:2229] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:08:14.041761 12505 raft_consensus.cc:2258] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:14.042060 12507 ts_tablet_manager.cc:1905] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:14.044134 12505 ts_tablet_manager.cc:1905] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:14.044535 12506 ts_tablet_manager.cc:1905] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:14.050094 12507 ts_tablet_manager.cc:1918] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:14.050290 12507 log.cc:1196] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/1c98cf87952b4160ae15a03ed18638cc
I20260811 03:08:14.050940 12507 ts_tablet_manager.cc:1939] T 1c98cf87952b4160ae15a03ed18638cc P 900519cb725642278519e3032efa2a6e: Deleting consensus metadata
I20260811 03:08:14.053341 11956 catalog_manager.cc:4841] TS 900519cb725642278519e3032efa2a6e (127.27.109.2:45895): tablet 1c98cf87952b4160ae15a03ed18638cc (table test-table [id=8f4fbff6fc1a47fcb1e59136e33c1fcf]) successfully deleted
I20260811 03:08:14.053933 12506 ts_tablet_manager.cc:1918] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:14.054629 12505 ts_tablet_manager.cc:1918] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260811 03:08:14.054747 11956 catalog_manager.cc:4808] TS 900519cb725642278519e3032efa2a6e (127.27.109.2:45895): delete failed for tablet 1c98cf87952b4160ae15a03ed18638cc because the tablet was not found. No further retry: Not found: Tablet not found: 1c98cf87952b4160ae15a03ed18638cc
I20260811 03:08:14.054911 12505 log.cc:1196] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/1c98cf87952b4160ae15a03ed18638cc
I20260811 03:08:14.055544 12506 log.cc:1196] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/1c98cf87952b4160ae15a03ed18638cc
I20260811 03:08:14.056233 12506 ts_tablet_manager.cc:1939] T 1c98cf87952b4160ae15a03ed18638cc P b07eb5cb1d9f415dbad0d7a7e0a6ba5a: Deleting consensus metadata
I20260811 03:08:14.056310 12505 ts_tablet_manager.cc:1939] T 1c98cf87952b4160ae15a03ed18638cc P def07bb84f0345a1baffd3df05e63670: Deleting consensus metadata
I20260811 03:08:14.058202 11957 catalog_manager.cc:4841] TS def07bb84f0345a1baffd3df05e63670 (127.27.109.1:43711): tablet 1c98cf87952b4160ae15a03ed18638cc (table test-table [id=8f4fbff6fc1a47fcb1e59136e33c1fcf]) successfully deleted
I20260811 03:08:14.058207 11954 catalog_manager.cc:4841] TS b07eb5cb1d9f415dbad0d7a7e0a6ba5a (127.27.109.3:34997): tablet 1c98cf87952b4160ae15a03ed18638cc (table test-table [id=8f4fbff6fc1a47fcb1e59136e33c1fcf]) successfully deleted
I20260811 03:08:14.067716 12478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:14.013449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.27.109.1:43711)
0811 03:08:14.013670 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:14.013677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:14.034256 (+ 20579us) server_negotiation.cc:407] Connection header received
0811 03:08:14.034504 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:14.034511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:14.034625 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:14.034805 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:14.039636 (+ 4831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:14.041158 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:14.054246 (+ 13088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:14.055151 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:14.055315 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:14.060039 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:14.060052 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:14.060066 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:14.060108 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:14.063668 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:14.064712 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:14.064721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:14.064723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:14.064986 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:14.065664 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:14.065670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:14.065674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:14.066238 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:14.066577 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:14.067063 (+ 486us) server_negotiation.cc:299] Negotiation successful
0811 03:08:14.067334 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 03:08:14.068521 12164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:14.180447 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 12009
I20260811 03:08:14.205794 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 12174
I20260811 03:08:14.221062 12477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:14.206667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.27.109.2:45895)
0811 03:08:14.206888 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:14.206893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:14.206913 (+ 20us) server_negotiation.cc:407] Connection header received
0811 03:08:14.207060 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:14.207065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:14.207152 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:14.207244 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:14.208686 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:14.209444 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:14.210947 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:14.211372 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:14.211469 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:14.214227 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:14.214243 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:14.214261 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:14.214309 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:14.217691 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:14.218477 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:14.218484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:14.218487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:14.218643 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:14.219253 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:14.219257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:14.219259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:14.219701 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:14.219939 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:14.220341 (+ 402us) server_negotiation.cc:299] Negotiation successful
0811 03:08:14.220542 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 03:08:14.221905 12314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:14.316156 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 12174
I20260811 03:08:14.337741 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 12323
I20260811 03:08:14.353509 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:14.338629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.27.109.3:34997)
0811 03:08:14.338877 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:14.338883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:14.338906 (+ 23us) server_negotiation.cc:407] Connection header received
0811 03:08:14.339178 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:14.339183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:14.339369 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:14.339517 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:14.341052 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:14.341789 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:14.343376 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:14.343786 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:14.343929 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:14.346778 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:14.346795 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:14.346812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:14.346870 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:14.350126 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:14.351065 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:14.351073 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:14.351076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:14.351249 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:14.351678 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:14.351682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:14.351684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:14.352162 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:14.352477 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:14.352898 (+ 421us) server_negotiation.cc:299] Negotiation successful
0811 03:08:14.353150 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260811 03:08:14.354290 12458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:14.447225 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 12323
I20260811 03:08:14.469877 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 11936
I20260811 03:08:14.471288 12002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:14.589407 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 11936
I20260811 03:08:14.617349 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32863
--webserver_interface=127.27.109.62
--webserver_port=43105
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:32863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:14.981400 12513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:14.981788 12513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:14.981954 12513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:14.989776 12513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:14.989907 12513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:14.989964 12513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:14.990025 12513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:14.990072 12513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:15.000681 12513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:32863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=43105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:15.002219 12513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:15.003705 12513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:15.018256 12522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:15.018353 12520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:15.018340 12519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:15.023857 12513 server_base.cc:1034] running on GCE node
I20260811 03:08:15.024807 12513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:15.026582 12513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:15.027878 12513 hybrid_clock.cc:648] HybridClock initialized: now 1786417695027808 us; error 59 us; skew 500 ppm
I20260811 03:08:15.028291 12513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:15 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417695, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:15.031630 12513 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:08:15.033365 12513 webserver.cc:460] Webserver started at http://127.27.109.62:43105/ using document root <none> and password file <none>
I20260811 03:08:15.034128 12513 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:15.034245 12513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:15.040408 12513 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:08:15.045372 12529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:15.047014 12513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 03:08:15.047180 12513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "836425ae2dee4a99b606933df3f7e0a6"
format_stamp: "Formatted at 2026-08-11 03:08:08 on dist-test-slave-8tdl"
server_key: "2460ed5a69e453d623a4f6c1421fe905"
server_key_iv: "100b93d8b89b9dd925c82d42fbc8f234"
server_key_version: "encryptionkey@0"
I20260811 03:08:15.048228 12513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:15.077510 12513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:15.082103 12513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:15.082433 12513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:15.102352 12513 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:32863
I20260811 03:08:15.102366 12586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:32863 every 8 connection(s)
I20260811 03:08:15.104002 12513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:08:15.110258 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 12513
I20260811 03:08:15.111364 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:43711
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:32863
--webserver_port=44451
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:08:15.111819 12587 sys_catalog.cc:263] Verifying existing consensus state
I20260811 03:08:15.117311 12587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Bootstrap starting.
I20260811 03:08:15.141171 12587 log.cc:824] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:15.158964 12587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 03:08:15.159600 12587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6: Bootstrap complete.
I20260811 03:08:15.169358 12587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:15.170321 12587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836425ae2dee4a99b606933df3f7e0a6, State: Initialized, Role: FOLLOWER
I20260811 03:08:15.171105 12587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:15.171365 12587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:08:15.171492 12587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:08:15.171712 12587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 2 FOLLOWER]: Advancing to term 3
I20260811 03:08:15.175997 12587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:15.176640 12587 leader_election.cc:302] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 836425ae2dee4a99b606933df3f7e0a6; no voters:
I20260811 03:08:15.177357 12587 leader_election.cc:288] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 03:08:15.177507 12592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 03:08:15.178797 12592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [term 3 LEADER]: Becoming Leader. State: Replica: 836425ae2dee4a99b606933df3f7e0a6, State: Running, Role: LEADER
I20260811 03:08:15.179411 12592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } }
I20260811 03:08:15.179823 12587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:08:15.182282 12594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836425ae2dee4a99b606933df3f7e0a6. Latest consensus state: current_term: 3 leader_uuid: "836425ae2dee4a99b606933df3f7e0a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } } }
I20260811 03:08:15.182740 12594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:15.182399 12593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "836425ae2dee4a99b606933df3f7e0a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "836425ae2dee4a99b606933df3f7e0a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32863 } } }
I20260811 03:08:15.182888 12593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836425ae2dee4a99b606933df3f7e0a6 [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:15.192749 12600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:08:15.202927 12600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:08:15.204568 12600 catalog_manager.cc:1259] Loaded cluster ID: 9b130beb2c3a402da9b24be0af2828cd
I20260811 03:08:15.204718 12600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260811 03:08:15.209306 12600 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fec5e5669ad google::LogMessage::Fail() at ??:0
@ 0x7fec5e56988a google::LogMessage::SendToLog() at ??:0
@ 0x7fec5e5664de google::LogMessage::Flush() at ??:0
@ 0x7fec5e567a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fec73f99512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fec73f96976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fec7401e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fec7401e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fec611b2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fec611b591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fec611b5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fec61190eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fec6446a6db start_thread at ??:0
@ 0x7fec5d40061f clone at ??:0
W20260811 03:08:15.575125 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:15.575438 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:15.575605 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:15.585126 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:15.585314 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:15.585510 12589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:08:15.596421 12589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:43711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=44451
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:15.597877 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:15.599406 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:15.613827 12632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:15.613847 12633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:15.613785 12636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:15.615068 12589 server_base.cc:1034] running on GCE node
I20260811 03:08:15.615832 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:15.616916 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:15.618168 12589 hybrid_clock.cc:648] HybridClock initialized: now 1786417695618105 us; error 54 us; skew 500 ppm
I20260811 03:08:15.618538 12589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:15 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417695, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:15.622027 12589 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:08:15.623680 12589 webserver.cc:460] Webserver started at http://127.27.109.1:44451/ using document root <none> and password file <none>
I20260811 03:08:15.624446 12589 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:15.624644 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:15.630420 12589 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 03:08:15.633383 12644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:15.635058 12589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 03:08:15.635203 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "def07bb84f0345a1baffd3df05e63670"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "39b20b899cd49795c10d4498a423f1a1"
server_key_iv: "1eacb5ea17329b23f83d55711a61e92c"
server_key_version: "encryptionkey@0"
I20260811 03:08:15.636272 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:15.648007 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:15.652727 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:15.653016 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:15.654111 12589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:15.655753 12589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:15.655869 12589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:15.655963 12589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:15.656030 12589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:15.694410 12589 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:43711
I20260811 03:08:15.694525 12763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:43711 every 8 connection(s)
I20260811 03:08:15.696213 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:08:15.698385 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 12589
I20260811 03:08:15.699152 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:45895
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:32863
--webserver_port=38303
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:08:15.702232 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:15.699767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:40069)
0811 03:08:15.700712 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:15.700991 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":363,"threads_started":1}
W20260811 03:08:15.703078 12764 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:32863 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:32863: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
I20260811 03:08:15.704156 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:15.703676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:42053)
0811 03:08:15.703876 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:15.703978 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 03:08:15.705169 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:15.704730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:51685)
0811 03:08:15.704904 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:15.705010 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 03:08:15.705472 12764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 03:08:16.063650 12767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:16.063954 12767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:16.064163 12767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:16.071808 12767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:16.071935 12767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:16.072166 12767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:08:16.082396 12767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:45895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=38303
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:16.083956 12767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:16.085388 12767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:16.100131 12775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:16.100215 12773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:16.100235 12772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:16.101433 12767 server_base.cc:1034] running on GCE node
I20260811 03:08:16.102135 12767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:16.103248 12767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:16.104494 12767 hybrid_clock.cc:648] HybridClock initialized: now 1786417696104437 us; error 48 us; skew 500 ppm
I20260811 03:08:16.104856 12767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:16 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417696, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:16.108325 12767 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:08:16.109897 12767 webserver.cc:460] Webserver started at http://127.27.109.2:38303/ using document root <none> and password file <none>
I20260811 03:08:16.110652 12767 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:16.110877 12767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:16.116642 12767 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:08:16.119941 12782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:16.121456 12767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 03:08:16.121670 12767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "900519cb725642278519e3032efa2a6e"
format_stamp: "Formatted at 2026-08-11 03:08:09 on dist-test-slave-8tdl"
server_key: "2933851f6e3be2a826c85d05ae441376"
server_key_iv: "2d60a2c94eb760022cb5d5ff243e4da5"
server_key_version: "encryptionkey@0"
I20260811 03:08:16.122804 12767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:16.159344 12767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:16.164157 12767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:16.164477 12767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:16.165618 12767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:16.167212 12767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:16.167354 12767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:16.167508 12767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:16.167603 12767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:16.203306 12767 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45895
I20260811 03:08:16.203395 12905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45895 every 8 connection(s)
I20260811 03:08:16.204988 12767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:08:16.210866 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:16.208399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:33739)
0811 03:08:16.209288 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:16.209571 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":378,"threads_started":1}
W20260811 03:08:16.211849 12906 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:32863 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:32863: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
I20260811 03:08:16.213006 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:16.212498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:48017)
0811 03:08:16.212726 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:16.212850 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 03:08:16.213383 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 12767
I20260811 03:08:16.214206 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:16.213752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:37097)
0811 03:08:16.213921 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:16.214031 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 03:08:16.214180 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:34997
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:32863
--webserver_port=40775
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:16.214485 12906 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 03:08:16.578189 12909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:16.578504 12909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:16.578692 12909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:16.586156 12909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:16.586280 12909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:16.586491 12909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:08:16.596740 12909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:34997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=40775
--tserver_master_addrs=127.27.109.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:16.598167 12909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:16.599557 12909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:16.613317 12917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:16.613395 12915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:16.613982 12914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:16.614359 12909 server_base.cc:1034] running on GCE node
I20260811 03:08:16.615082 12909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:16.616065 12909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:16.617282 12909 hybrid_clock.cc:648] HybridClock initialized: now 1786417696617242 us; error 28 us; skew 500 ppm
I20260811 03:08:16.617651 12909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:16 dist-test-slave-8tdl krb5kdc[11316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417696, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:16.621286 12909 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:08:16.622927 12909 webserver.cc:460] Webserver started at http://127.27.109.3:40775/ using document root <none> and password file <none>
I20260811 03:08:16.623647 12909 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:16.623811 12909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:16.629855 12909 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 03:08:16.632905 12925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:16.634353 12909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:08:16.634552 12909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "b07eb5cb1d9f415dbad0d7a7e0a6ba5a"
format_stamp: "Formatted at 2026-08-11 03:08:10 on dist-test-slave-8tdl"
server_key: "570764b973a110cc615eaa1a25368afa"
server_key_iv: "7cac46592ca33d72911e883f731c8577"
server_key_version: "encryptionkey@0"
I20260811 03:08:16.635799 12909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:16.651611 12909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:16.655933 12909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:16.656247 12909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:16.657323 12909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:16.658831 12909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:16.658955 12909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:16.659101 12909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:16.659204 12909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:16.694981 12909 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:34997
I20260811 03:08:16.694999 13048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:34997 every 8 connection(s)
I20260811 03:08:16.697077 12909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:08:16.702194 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:16.700359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:47371)
0811 03:08:16.701153 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:16.701362 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":306,"threads_started":1}
W20260811 03:08:16.702970 13049 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:32863 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:32863: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
I20260811 03:08:16.703814 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:16.703500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:33705)
0811 03:08:16.703641 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:16.703708 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260811 03:08:16.704674 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:16.704330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:47749)
0811 03:08:16.704474 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:16.704565 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 03:08:16.704918 13049 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 03:08:16.706661 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:16.705931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:55287)
0811 03:08:16.706360 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:16.706446 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":199,"threads_started":1}
I20260811 03:08:16.706766 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 12909
I20260811 03:08:17.216384 13055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:17.215179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:35151)
0811 03:08:17.215855 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:17.216015 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":313,"threads_started":1}
I20260811 03:08:17.706496 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:17.705616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:44789)
0811 03:08:17.706129 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:17.706257 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":256,"threads_started":1}
I20260811 03:08:17.708453 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:17.707672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:33549)
0811 03:08:17.708157 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:17.708245 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":228,"threads_started":1}
I20260811 03:08:18.218631 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:18.217521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:58781)
0811 03:08:18.218239 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:18.218373 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":393,"threads_started":1}
I20260811 03:08:18.708568 13060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:18.707632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:53267)
0811 03:08:18.708171 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:18.708273 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":257,"threads_started":1}
I20260811 03:08:18.710177 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:18.709487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:58795)
0811 03:08:18.709923 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:18.710006 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":237,"threads_started":1}
I20260811 03:08:19.220482 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:19.219721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:49859)
0811 03:08:19.220204 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:19.220298 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":231,"threads_started":1}
I20260811 03:08:19.710278 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:19.709502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:44187)
0811 03:08:19.709970 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:19.710097 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":236,"threads_started":1}
I20260811 03:08:19.711918 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:19.711144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:42421)
0811 03:08:19.711641 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:19.711721 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
I20260811 03:08:20.222764 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:20.221598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:43659)
0811 03:08:20.222388 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:20.222516 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":351,"threads_started":1}
I20260811 03:08:20.712234 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:20.711439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:41611)
0811 03:08:20.711926 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:20.712018 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":232,"threads_started":1}
I20260811 03:08:20.713603 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:20.712902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:37459)
0811 03:08:20.713335 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:20.713439 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260811 03:08:21.224834 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:21.223829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:56189)
0811 03:08:21.224436 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:21.224541 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":325,"threads_started":1}
I20260811 03:08:21.714443 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:21.713513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:47059)
0811 03:08:21.714106 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:21.714238 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":305,"threads_started":1}
I20260811 03:08:21.715337 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:21.714450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:38859)
0811 03:08:21.715083 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:21.715177 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":395,"threads_started":1}
I20260811 03:08:22.226838 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:22.225982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:36073)
0811 03:08:22.226477 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:22.226585 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260811 03:08:22.716308 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:22.715404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:55149)
0811 03:08:22.715925 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:22.716075 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":249,"threads_started":1}
I20260811 03:08:22.717197 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:22.716337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:39641)
0811 03:08:22.716888 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:22.716982 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
I20260811 03:08:23.228730 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:23.227812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:60995)
0811 03:08:23.228410 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:23.228506 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":317,"threads_started":1}
I20260811 03:08:23.719094 13077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:23.717501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:36271)
0811 03:08:23.718756 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:23.718859 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1087,"thread_start_us":294,"threads_started":1}
I20260811 03:08:23.719076 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:23.718259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:44433)
0811 03:08:23.718821 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:23.718912 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":284,"threads_started":1}
I20260811 03:08:24.230646 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:24.229725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:43209)
0811 03:08:24.230373 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:24.230461 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":316,"threads_started":1}
I20260811 03:08:24.721277 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:24.720248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:53251)
0811 03:08:24.720940 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:24.721051 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":388,"threads_started":1}
I20260811 03:08:24.721338 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:24.720290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:41429)
0811 03:08:24.721051 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:24.721147 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":493,"threads_started":1}
I20260811 03:08:25.232380 13082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:25.231562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:47643)
0811 03:08:25.232074 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:25.232199 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":246,"threads_started":1}
I20260811 03:08:25.726051 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:25.722382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:58093)
0811 03:08:25.725689 (+ 3307us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:25.725804 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3144,"thread_start_us":2988,"threads_started":1}
I20260811 03:08:25.726967 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:25.722401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:34321)
0811 03:08:25.726602 (+ 4201us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:25.726734 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4082,"thread_start_us":3983,"threads_started":1}
I20260811 03:08:26.234127 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:26.233348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:45921)
0811 03:08:26.233843 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:26.233943 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":282,"threads_started":1}
I20260811 03:08:26.728253 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:26.727263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:46635)
0811 03:08:26.727958 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:26.728069 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":379,"threads_started":1}
I20260811 03:08:26.728936 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:26.728066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:53811)
0811 03:08:26.728698 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:26.728776 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":275,"threads_started":1}
I20260811 03:08:27.235968 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:27.235164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:38519)
0811 03:08:27.235683 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:27.235776 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":251,"threads_started":1}
I20260811 03:08:27.730109 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:27.729284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:36923)
0811 03:08:27.729812 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:27.729908 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":251,"threads_started":1}
I20260811 03:08:27.730969 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:27.729924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:42291)
0811 03:08:27.730724 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:27.730819 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":294,"threads_started":1}
I20260811 03:08:28.237673 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:28.236913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:50191)
0811 03:08:28.237416 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:28.237502 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
I20260811 03:08:28.732153 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:28.731263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:44209)
0811 03:08:28.731875 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:28.731978 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":355,"threads_started":1}
I20260811 03:08:28.732811 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:28.731867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:58797)
0811 03:08:28.732487 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:28.732644 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":341,"threads_started":1}
I20260811 03:08:29.239941 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:29.238864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:58147)
0811 03:08:29.239523 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:29.239620 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":351,"threads_started":1}
I20260811 03:08:29.734337 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:29.733320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:41199)
0811 03:08:29.734019 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:29.734132 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":395,"threads_started":1}
I20260811 03:08:29.734818 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:29.733781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:55823)
0811 03:08:29.734412 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:29.734556 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":369,"threads_started":1}
I20260811 03:08:30.242033 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:30.241138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:37307)
0811 03:08:30.241726 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:30.241820 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":290,"threads_started":1}
I20260811 03:08:30.736516 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:30.735502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:33721)
0811 03:08:30.736101 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:30.736225 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":250,"threads_started":1}
I20260811 03:08:30.737098 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:30.736176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:48289)
0811 03:08:30.736829 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:30.736931 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":262,"threads_started":1}
I20260811 03:08:31.243990 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:31.243238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.2:60283)
0811 03:08:31.243701 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:31.243793 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260811 03:08:31.714510 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 12589
I20260811 03:08:31.738759 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:31.737761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.3:42647)
0811 03:08:31.738320 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:31.738452 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":304,"threads_started":1}
I20260811 03:08:31.739288 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:31.738068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32863 (local address 127.27.109.1:60993)
0811 03:08:31.738698 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:31.738954 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":296,"threads_started":1}
I20260811 03:08:31.743386 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:31.715925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.27.109.1:43711)
0811 03:08:31.716627 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:31.716643 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:31.716681 (+ 38us) server_negotiation.cc:407] Connection header received
0811 03:08:31.718327 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:31.718337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:31.718857 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:31.719163 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:31.720859 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:31.723173 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:31.725280 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:31.726441 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:31.726721 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:31.735586 (+ 8865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:31.735595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:31.735614 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:31.735662 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:31.739868 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:31.740755 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:31.740763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:31.740770 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:31.740956 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:31.741440 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:31.741444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:31.741446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:31.742025 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:31.742275 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:31.742663 (+ 388us) server_negotiation.cc:299] Negotiation successful
0811 03:08:31.743112 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":319,"threads_started":1}
I20260811 03:08:31.744984 12759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:31.832010 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 12589
I20260811 03:08:31.856690 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 12767
I20260811 03:08:31.876965 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:31.857883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.27.109.2:45895)
0811 03:08:31.858570 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:31.858586 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:31.858625 (+ 39us) server_negotiation.cc:407] Connection header received
0811 03:08:31.860541 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:31.860552 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:31.861017 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:31.861333 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:31.863226 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:31.864658 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:31.866060 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:31.866954 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:31.867110 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:31.870075 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:31.870084 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:31.870101 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:31.870147 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:31.873234 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:31.874059 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:31.874066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:31.874073 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:31.874256 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:31.874747 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:31.874755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:31.874760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:31.875394 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:31.875656 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:31.876158 (+ 502us) server_negotiation.cc:299] Negotiation successful
0811 03:08:31.876645 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":354,"threads_started":1}
I20260811 03:08:31.878468 12899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:31.964968 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 12767
I20260811 03:08:31.988051 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 12909
I20260811 03:08:32.007376 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:31.989117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.27.109.3:34997)
0811 03:08:31.989696 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:31.989712 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:31.989754 (+ 42us) server_negotiation.cc:407] Connection header received
0811 03:08:31.991570 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:31.991589 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:31.992033 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:31.992336 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:31.993872 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:31.995447 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:31.996774 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:31.997453 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:31.997622 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:32.000560 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:32.000568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:32.000604 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:32.000647 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:32.003891 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:32.004709 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:32.004718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:32.004725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:32.004952 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:32.005340 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:32.005343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:32.005345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:32.005938 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:32.006206 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:32.006598 (+ 392us) server_negotiation.cc:299] Negotiation successful
0811 03:08:32.007047 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":254,"threads_started":1}
I20260811 03:08:32.009030 13043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:32.097689 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 12909
2026-08-11T03:08:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25479 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:08:32 dist-test-slave-8tdl krb5kdc[13115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:08:32 dist-test-slave-8tdl krb5kdc[13115](info): set up 2 sockets
Aug 11 03:08:32 dist-test-slave-8tdl krb5kdc[13115](info): commencing operation
krb5kdc: starting...
W20260811 03:08:33.360268 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:08:33 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:08:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:08:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:08:33.698513 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41965
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:34.069380 13131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:34.069711 13131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:34.069876 13131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:34.077535 13131 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 03:08:34.077656 13131 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 03:08:34.077694 13131 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 03:08:34.077807 13131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:34.077917 13131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:34.078034 13131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:34.078116 13131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:34.078202 13131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:34.088580 13131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:34.090080 13131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:34.091589 13131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:34.104576 13137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:34.104606 13138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:34.104573 13140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:34.105755 13131 server_base.cc:1034] running on GCE node
I20260811 03:08:34.106828 13131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:34.108386 13131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:34.109628 13131 hybrid_clock.cc:648] HybridClock initialized: now 1786417714109541 us; error 58 us; skew 500 ppm
I20260811 03:08:34.110037 13131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:34 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417714, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:34.113739 13131 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:08:34.115381 13131 webserver.cc:460] Webserver started at http://127.27.109.62:44181/ using document root <none> and password file <none>
I20260811 03:08:34.116078 13131 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:34.116206 13131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:34.116676 13131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:34.119885 13131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "3352b7df0d7f4338bb37b5ca7a28defe"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "4fdbfe19c22ded3d64e4a060efa53185"
server_key_iv: "3847231c39e8ccb8767d7ceef01cb62f"
server_key_version: "encryptionkey@0"
I20260811 03:08:34.120738 13131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "3352b7df0d7f4338bb37b5ca7a28defe"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "4fdbfe19c22ded3d64e4a060efa53185"
server_key_iv: "3847231c39e8ccb8767d7ceef01cb62f"
server_key_version: "encryptionkey@0"
I20260811 03:08:34.126221 13131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:08:34.130066 13148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:34.131582 13131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:08:34.131770 13131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "3352b7df0d7f4338bb37b5ca7a28defe"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "4fdbfe19c22ded3d64e4a060efa53185"
server_key_iv: "3847231c39e8ccb8767d7ceef01cb62f"
server_key_version: "encryptionkey@0"
I20260811 03:08:34.132009 13131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:34.169447 13131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:34.174386 13131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:34.174749 13131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:34.196285 13131 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41965
I20260811 03:08:34.196305 13203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41965 every 8 connection(s)
I20260811 03:08:34.197757 13131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:08:34.201726 13204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:34.202546 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 13131
I20260811 03:08:34.202813 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:08:34.203495 28084 external_mini_cluster.cc:1428] Setting key 65f1d433e807c7174ece8a4ac58f1baf
I20260811 03:08:34.216884 13204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Bootstrap starting.
Aug 11 03:08:34 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:34.220844 13204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:34.221954 13204 log.cc:824] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:34.225152 13204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: No bootstrap required, opened a new log
I20260811 03:08:34.228958 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:34.205703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.27.109.62:41965)
0811 03:08:34.206608 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:34.206623 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:34.206666 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:08:34.208448 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:34.208478 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:34.208898 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:34.209256 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:34.211167 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:34.212521 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:34.214809 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:34.215636 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:34.215782 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:34.220860 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:34.220871 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:34.220887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:34.220954 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:34.224392 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:34.225207 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:34.225218 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:34.225229 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:34.225441 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:34.225871 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:34.225876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:34.225881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:34.226447 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:34.226813 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:34.227351 (+ 538us) server_negotiation.cc:299] Negotiation successful
0811 03:08:34.227829 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":415,"threads_started":1}
I20260811 03:08:34.234522 13204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:08:34.235006 13204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:34.235134 13204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3352b7df0d7f4338bb37b5ca7a28defe, State: Initialized, Role: FOLLOWER
I20260811 03:08:34.235766 13204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:08:34.236050 13204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:08:34.236163 13204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:08:34.236297 13204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:34.239195 13204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:08:34.239753 13204 leader_election.cc:302] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3352b7df0d7f4338bb37b5ca7a28defe; no voters:
I20260811 03:08:34.240377 13204 leader_election.cc:288] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:08:34.240514 13209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:08:34.241977 13209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 1 LEADER]: Becoming Leader. State: Replica: 3352b7df0d7f4338bb37b5ca7a28defe, State: Running, Role: LEADER
I20260811 03:08:34.242496 13209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:08:34.242901 13204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:08:34.245564 13210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } } }
I20260811 03:08:34.245559 13211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3352b7df0d7f4338bb37b5ca7a28defe. Latest consensus state: current_term: 1 leader_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } } }
I20260811 03:08:34.246104 13210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:34.246100 13211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:34.247771 13215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:08:34.252640 13215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:08:34.263916 13215 catalog_manager.cc:1347] Generated new cluster ID: cbdf9e92667f4fd9979214722d56aef4
I20260811 03:08:34.264101 13215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:08:34.277076 13215 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:08:34.279148 13215 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:08:34.292505 13215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Generated new TSK 0
I20260811 03:08:34.293630 13215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:08:34.379387 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41965
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:34.774066 13232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:34.774492 13232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:34.774610 13232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:34.782428 13232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:34.782579 13232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:34.782775 13232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:08:34.793866 13232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:34.795415 13232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:34.796881 13232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:34.811858 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:34.811800 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:34.812179 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:34.812796 13232 server_base.cc:1034] running on GCE node
I20260811 03:08:34.813405 13232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:34.814479 13232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:34.815704 13232 hybrid_clock.cc:648] HybridClock initialized: now 1786417714815654 us; error 94 us; skew 500 ppm
I20260811 03:08:34.816069 13232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:34 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417714, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:34.819444 13232 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:08:34.821115 13232 webserver.cc:460] Webserver started at http://127.27.109.1:36937/ using document root <none> and password file <none>
I20260811 03:08:34.821805 13232 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:34.821916 13232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:34.822184 13232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:34.825438 13232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "546615372ca340ad8326852c668e837e"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "9e0540de41198c17bc45fc8989c32a3a"
server_key_iv: "367a8104a18f7fae04b296c93d9810fd"
server_key_version: "encryptionkey@0"
I20260811 03:08:34.826231 13232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "546615372ca340ad8326852c668e837e"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "9e0540de41198c17bc45fc8989c32a3a"
server_key_iv: "367a8104a18f7fae04b296c93d9810fd"
server_key_version: "encryptionkey@0"
I20260811 03:08:34.831827 13232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260811 03:08:34.835831 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:34.837631 13232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:08:34.837844 13232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "546615372ca340ad8326852c668e837e"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "9e0540de41198c17bc45fc8989c32a3a"
server_key_iv: "367a8104a18f7fae04b296c93d9810fd"
server_key_version: "encryptionkey@0"
I20260811 03:08:34.838140 13232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:34.855852 13232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:34.860924 13232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:34.861264 13232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:34.862351 13232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:34.863915 13232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:34.863992 13232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:34.864130 13232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:34.864199 13232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:34.900761 13232 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37181
I20260811 03:08:34.900810 13371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37181 every 8 connection(s)
I20260811 03:08:34.902171 13232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:08:34.908790 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 13232
I20260811 03:08:34.909044 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:08:34.909874 28084 external_mini_cluster.cc:1428] Setting key b42f6af46b33a63d966fd6a3a3e90010
Aug 11 03:08:34 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417714, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:34.928679 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:34.905262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:42771 (local address 127.27.109.62:41965)
0811 03:08:34.905760 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:34.905766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:34.906964 (+ 1198us) server_negotiation.cc:407] Connection header received
0811 03:08:34.909059 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:34.909069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:34.909187 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:34.909332 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:34.911421 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:34.912573 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:34.914888 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:34.915652 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:34.915814 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:34.920615 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:34.920624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:34.920631 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:34.920680 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:34.924413 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:34.925636 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:34.925645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:34.925651 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:34.925851 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:34.926516 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:34.926522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:34.926525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:34.926925 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:34.927107 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:34.927943 (+ 836us) server_negotiation.cc:299] Negotiation successful
0811 03:08:34.928230 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260811 03:08:34.929842 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:34.905889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:42771)
0811 03:08:34.906664 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:34.906776 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:34.908686 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:34.909560 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:34.909569 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:34.910303 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:34.911199 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:34.911228 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:34.912769 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:34.912774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:34.914654 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:34.914669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:34.915891 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:34.915896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:34.916124 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:34.916903 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:34.916940 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:34.920281 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:34.924648 (+ 4367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:34.924662 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:34.924687 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:34.925449 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:34.926021 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:34.926029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:34.926035 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:34.926356 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:34.927080 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:34.927089 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:34.927509 (+ 420us) client_negotiation.cc:769] Sending connection context
0811 03:08:34.927820 (+ 311us) client_negotiation.cc:240] Negotiation successful
0811 03:08:34.928470 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":299,"threads_started":1}
I20260811 03:08:34.931882 13372 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41965
I20260811 03:08:34.932264 13372 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:34.933266 13372 heartbeater.cc:511] Master 127.27.109.62:41965 requested a full tablet report, sending...
I20260811 03:08:34.937024 13166 ts_manager.cc:194] Registered new tserver with Master: 546615372ca340ad8326852c668e837e (127.27.109.1:37181)
I20260811 03:08:34.939512 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:42771
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:08:34.986918 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41965
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:35.345985 13382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:35.346309 13382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:35.346464 13382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:35.354249 13382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:35.354388 13382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:35.354607 13382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:08:35.365106 13382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:35.366541 13382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:35.368052 13382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:35.381992 13391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:35.382019 13389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:35.382521 13388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:35.382910 13382 server_base.cc:1034] running on GCE node
I20260811 03:08:35.383544 13382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:35.384557 13382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:35.385764 13382 hybrid_clock.cc:648] HybridClock initialized: now 1786417715385722 us; error 33 us; skew 500 ppm
I20260811 03:08:35.386070 13382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:35 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417715, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:35.389369 13382 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:08:35.391069 13382 webserver.cc:460] Webserver started at http://127.27.109.2:41917/ using document root <none> and password file <none>
I20260811 03:08:35.391812 13382 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:35.391950 13382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:35.392293 13382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:35.395579 13382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "60ba0342b9c045f68eed2932060444de"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "3b2632bb99eafcff8f03fcda983257b2"
server_key_iv: "ef9c57ed3d88be5396a33e48d5b22385"
server_key_version: "encryptionkey@0"
I20260811 03:08:35.396451 13382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "60ba0342b9c045f68eed2932060444de"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "3b2632bb99eafcff8f03fcda983257b2"
server_key_iv: "ef9c57ed3d88be5396a33e48d5b22385"
server_key_version: "encryptionkey@0"
I20260811 03:08:35.402323 13382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:08:35.406301 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:35.407899 13382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:08:35.408106 13382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "60ba0342b9c045f68eed2932060444de"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "3b2632bb99eafcff8f03fcda983257b2"
server_key_iv: "ef9c57ed3d88be5396a33e48d5b22385"
server_key_version: "encryptionkey@0"
I20260811 03:08:35.408349 13382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:35.433001 13382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:35.438014 13382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:35.438344 13382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:35.439603 13382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:35.441054 13382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:35.441135 13382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:35.441294 13382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:35.441349 13382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:35.477752 13382 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43893
I20260811 03:08:35.477815 13520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43893 every 8 connection(s)
I20260811 03:08:35.479193 13382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:08:35.480600 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 13382
I20260811 03:08:35.480808 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:08:35.481436 28084 external_mini_cluster.cc:1428] Setting key 110c1891b3c0d6d5a529d6f0b2187d98
Aug 11 03:08:35 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417715, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:35.504515 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:35.482346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:35727 (local address 127.27.109.62:41965)
0811 03:08:35.482881 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:35.482886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:35.484224 (+ 1338us) server_negotiation.cc:407] Connection header received
0811 03:08:35.486223 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:35.486232 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:35.486332 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:35.486456 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:35.488138 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:35.489376 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:35.491075 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:35.491865 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:35.492014 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:35.496849 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:35.496856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:35.496862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:35.496915 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:35.500803 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:35.501654 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:35.501661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:35.501667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:35.501843 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:35.502303 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:35.502311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:35.502316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:35.502791 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:35.502975 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:35.503690 (+ 715us) server_negotiation.cc:299] Negotiation successful
0811 03:08:35.504005 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":283,"threads_started":1}
I20260811 03:08:35.505187 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:35.483063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:35727)
0811 03:08:35.484033 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:35.484093 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:35.485919 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:35.486729 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:35.486747 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:35.487320 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:35.487935 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:35.487962 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:35.489542 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:35.489546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:35.490919 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:35.490932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:35.492503 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:35.492516 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:35.492742 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:35.493537 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:35.493562 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:35.496538 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:35.500976 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:35.500984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:35.500999 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:35.501506 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:35.501958 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:35.501961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:35.501963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:35.502175 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:35.502927 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:35.502934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:35.503298 (+ 364us) client_negotiation.cc:769] Sending connection context
0811 03:08:35.503536 (+ 238us) client_negotiation.cc:240] Negotiation successful
0811 03:08:35.503950 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":432,"threads_started":1}
I20260811 03:08:35.506919 13521 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41965
I20260811 03:08:35.507257 13521 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:35.508023 13521 heartbeater.cc:511] Master 127.27.109.62:41965 requested a full tablet report, sending...
I20260811 03:08:35.509642 13166 ts_manager.cc:194] Registered new tserver with Master: 60ba0342b9c045f68eed2932060444de (127.27.109.2:43893)
I20260811 03:08:35.510504 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:35727
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:08:35.555917 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41965
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:35.920768 13527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:35.921074 13527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:35.921175 13527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:35.928628 13527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:35.928748 13527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:35.928864 13527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:08:35.939363 13527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:35.940717 13527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:35.942045 13527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 03:08:35.944263 13372 heartbeater.cc:503] Master 127.27.109.62:41965 was elected leader, sending a full tablet report...
W20260811 03:08:35.956429 13535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:35.956519 13532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:35.956506 13533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:35.960361 13527 server_base.cc:1034] running on GCE node
I20260811 03:08:35.961163 13527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:35.962246 13527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:35.963518 13527 hybrid_clock.cc:648] HybridClock initialized: now 1786417715963455 us; error 68 us; skew 500 ppm
I20260811 03:08:35.963891 13527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:35 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417715, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:35.967451 13527 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:08:35.969228 13527 webserver.cc:460] Webserver started at http://127.27.109.3:40835/ using document root <none> and password file <none>
I20260811 03:08:35.970079 13527 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:35.970280 13527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:35.970642 13527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:08:35.974056 13527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "1fd7bd27874f4700a0fc11658f040935"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "411f7a672f57e8f870222fe5512daa96"
server_key_iv: "7af77c0c4995df10feaa3497057b57bc"
server_key_version: "encryptionkey@0"
I20260811 03:08:35.975042 13527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "1fd7bd27874f4700a0fc11658f040935"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "411f7a672f57e8f870222fe5512daa96"
server_key_iv: "7af77c0c4995df10feaa3497057b57bc"
server_key_version: "encryptionkey@0"
I20260811 03:08:35.981362 13527 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:08:35.985571 13542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:35.987164 13527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:08:35.987370 13527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "1fd7bd27874f4700a0fc11658f040935"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "411f7a672f57e8f870222fe5512daa96"
server_key_iv: "7af77c0c4995df10feaa3497057b57bc"
server_key_version: "encryptionkey@0"
I20260811 03:08:35.987646 13527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:36.015358 13527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:36.020955 13527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:36.021340 13527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:36.022454 13527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:36.023976 13527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:36.024091 13527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:36.024246 13527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:36.024322 13527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:36.060243 13527 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:42219
I20260811 03:08:36.060302 13663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:42219 every 8 connection(s)
I20260811 03:08:36.061599 13527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:08:36.070247 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 13527
I20260811 03:08:36.070453 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:08:36.071004 28084 external_mini_cluster.cc:1428] Setting key 6b35504d057dc2d25a0805cf7b0780bc
Aug 11 03:08:36 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417715, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:36.082909 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.064295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:39721 (local address 127.27.109.62:41965)
0811 03:08:36.064836 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.064843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.065817 (+ 974us) server_negotiation.cc:407] Connection header received
0811 03:08:36.067854 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.067859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.067936 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.068037 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:36.069767 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.070785 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.072637 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.073053 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.073214 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.077059 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:36.077076 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:36.077079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:36.077115 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:36.079910 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.080689 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.080693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.080696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.080827 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.081246 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.081249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.081250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.081509 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:36.081630 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.082343 (+ 713us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.082542 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":264,"threads_started":1}
I20260811 03:08:36.083985 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.064829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:39721)
0811 03:08:36.065624 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.065673 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:36.067553 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:36.068302 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:36.068311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:36.068919 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:36.069574 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.069595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.070951 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.070955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.072478 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.072491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.073315 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.073323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.073620 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.074409 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:36.074442 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:36.076828 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:36.080085 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:36.080093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:36.080107 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:36.080581 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:36.080931 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:36.080934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:36.080937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:36.081149 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:36.081649 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:36.081656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:36.081997 (+ 341us) client_negotiation.cc:769] Sending connection context
0811 03:08:36.082265 (+ 268us) client_negotiation.cc:240] Negotiation successful
0811 03:08:36.082757 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":291,"threads_started":1}
I20260811 03:08:36.085618 13664 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41965
I20260811 03:08:36.085951 13664 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:36.086745 13664 heartbeater.cc:511] Master 127.27.109.62:41965 requested a full tablet report, sending...
I20260811 03:08:36.088243 13166 ts_manager.cc:194] Registered new tserver with Master: 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219)
I20260811 03:08:36.089036 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:39721
I20260811 03:08:36.099375 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:08:36.117928 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.104162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.27.109.62:41965)
0811 03:08:36.104376 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.104385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.104768 (+ 383us) server_negotiation.cc:407] Connection header received
0811 03:08:36.104994 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.104998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.105076 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.105181 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:36.106718 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.107460 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.108874 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.109335 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.109470 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.112202 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:36.112207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:36.112210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:36.112244 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:36.115059 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.115811 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.115818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.115821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.115967 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.116351 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.116359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.116362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.116713 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:36.116987 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.117348 (+ 361us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.117593 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:08:36.122299 13166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:08:36.129590 13166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:08:36.164774 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.143671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.27.109.3:42219)
0811 03:08:36.155583 (+ 11912us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.155594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.155639 (+ 45us) server_negotiation.cc:407] Connection header received
0811 03:08:36.155751 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.155760 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.156050 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.156275 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:36.157595 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.159314 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.161749 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.163137 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.163269 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.163521 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.164169 (+ 648us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.164380 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11722,"thread_start_us":11567,"threads_started":1}
I20260811 03:08:36.164805 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.142988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:42219 (local address 127.0.0.1:34224)
0811 03:08:36.155203 (+ 12215us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.155228 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:36.155382 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:36.156618 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:36.156627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:36.156661 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:36.157362 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.157378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.159476 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.159481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.161578 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.161591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.163665 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.163672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.163875 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.163942 (+ 67us) client_negotiation.cc:769] Sending connection context
0811 03:08:36.164094 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 03:08:36.164320 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12046,"mutex_wait_us":106,"thread_start_us":11849,"threads_started":1}
I20260811 03:08:36.165433 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.148726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.27.109.2:43893)
0811 03:08:36.154506 (+ 5780us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.154520 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.156156 (+ 1636us) server_negotiation.cc:407] Connection header received
0811 03:08:36.156267 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.156274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.156504 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.156730 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:36.158073 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.159653 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.162291 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.163798 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.163933 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.164132 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.164908 (+ 776us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.165086 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5545,"thread_start_us":5386,"threads_started":1}
I20260811 03:08:36.165825 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.142981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.27.109.1:37181)
0811 03:08:36.156555 (+ 13574us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.156564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.156589 (+ 25us) server_negotiation.cc:407] Connection header received
0811 03:08:36.156680 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.156688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.156909 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.157134 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:36.158477 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.160121 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.162670 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.164193 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.164330 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.164545 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.165386 (+ 841us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.165566 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13403,"thread_start_us":13243,"threads_started":1}
I20260811 03:08:36.166385 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.148077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43893 (local address 127.0.0.1:44644)
0811 03:08:36.155872 (+ 7795us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.155893 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:36.156018 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:36.156727 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:36.156732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:36.156759 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:36.157739 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.157755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.159790 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.159796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.162113 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.162130 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.164279 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.164285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.164459 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.164494 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 03:08:36.164595 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 03:08:36.164749 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7672,"mutex_wait_us":29,"thread_start_us":4529,"threads_started":1}
I20260811 03:08:36.166819 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.142820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:37920)
0811 03:08:36.153176 (+ 10356us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.153237 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:36.153489 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:36.157149 (+ 3660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:36.157155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:36.157187 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:36.158279 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.158293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.160313 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.160325 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.162512 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.162522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.164726 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.164734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.164942 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.164991 (+ 49us) client_negotiation.cc:769] Sending connection context
0811 03:08:36.165078 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 03:08:36.165204 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9992,"mutex_wait_us":22,"thread_start_us":12901,"threads_started":1}
I20260811 03:08:36.169885 13301 tablet_service.cc:1467] Processing CreateTablet for tablet 43236bb0717d4e49b395d28e2956ac3f (DEFAULT_TABLE table=test-table [id=4e0ee1d07b6d47618adf4646f436821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:36.169479 13592 tablet_service.cc:1467] Processing CreateTablet for tablet 43236bb0717d4e49b395d28e2956ac3f (DEFAULT_TABLE table=test-table [id=4e0ee1d07b6d47618adf4646f436821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:36.170282 13449 tablet_service.cc:1467] Processing CreateTablet for tablet 43236bb0717d4e49b395d28e2956ac3f (DEFAULT_TABLE table=test-table [id=4e0ee1d07b6d47618adf4646f436821c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:36.171339 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43236bb0717d4e49b395d28e2956ac3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:36.171609 13592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43236bb0717d4e49b395d28e2956ac3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:36.171622 13449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43236bb0717d4e49b395d28e2956ac3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:36.183223 13684 tablet_bootstrap.cc:492] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Bootstrap starting.
I20260811 03:08:36.187002 13684 tablet_bootstrap.cc:654] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:36.188402 13684 log.cc:824] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:36.190052 13685 tablet_bootstrap.cc:492] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Bootstrap starting.
I20260811 03:08:36.191766 13686 tablet_bootstrap.cc:492] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Bootstrap starting.
I20260811 03:08:36.192076 13684 tablet_bootstrap.cc:492] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: No bootstrap required, opened a new log
I20260811 03:08:36.192404 13684 ts_tablet_manager.cc:1397] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260811 03:08:36.195814 13686 tablet_bootstrap.cc:654] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:36.197242 13686 log.cc:824] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:36.201777 13686 tablet_bootstrap.cc:492] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: No bootstrap required, opened a new log
I20260811 03:08:36.201823 13685 tablet_bootstrap.cc:654] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:36.202242 13686 ts_tablet_manager.cc:1397] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260811 03:08:36.203424 13685 log.cc:824] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:36.203785 13684 raft_consensus.cc:348] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.204505 13684 raft_consensus.cc:374] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:36.204722 13684 raft_consensus.cc:729] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546615372ca340ad8326852c668e837e, State: Initialized, Role: FOLLOWER
I20260811 03:08:36.205824 13684 consensus_queue.cc:260] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.214359 13685 tablet_bootstrap.cc:492] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: No bootstrap required, opened a new log
I20260811 03:08:36.214788 13685 ts_tablet_manager.cc:1397] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.000s
I20260811 03:08:36.214807 13684 ts_tablet_manager.cc:1428] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Time spent starting tablet: real 0.022s user 0.015s sys 0.008s
I20260811 03:08:36.215070 13686 raft_consensus.cc:348] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.215554 13686 raft_consensus.cc:374] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:36.215677 13686 raft_consensus.cc:729] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd7bd27874f4700a0fc11658f040935, State: Initialized, Role: FOLLOWER
I20260811 03:08:36.216320 13686 consensus_queue.cc:260] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.218580 13664 heartbeater.cc:503] Master 127.27.109.62:41965 was elected leader, sending a full tablet report...
I20260811 03:08:36.219292 13686 ts_tablet_manager.cc:1428] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260811 03:08:36.224887 13685 raft_consensus.cc:348] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.225302 13685 raft_consensus.cc:374] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:36.225428 13685 raft_consensus.cc:729] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba0342b9c045f68eed2932060444de, State: Initialized, Role: FOLLOWER
I20260811 03:08:36.226054 13685 consensus_queue.cc:260] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.227246 13521 heartbeater.cc:503] Master 127.27.109.62:41965 was elected leader, sending a full tablet report...
I20260811 03:08:36.228001 13685 ts_tablet_manager.cc:1428] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260811 03:08:36.231377 13522 tablet.cc:2369] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:36.231680 13693 raft_consensus.cc:482] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:08:36.231957 13693 raft_consensus.cc:504] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.233582 13693 leader_election.cc:288] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 546615372ca340ad8326852c668e837e (127.27.109.1:37181), 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219)
I20260811 03:08:36.236075 13691 raft_consensus.cc:482] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:08:36.236382 13691 raft_consensus.cc:504] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.238133 13691 leader_election.cc:288] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219), 60ba0342b9c045f68eed2932060444de (127.27.109.2:43893)
I20260811 03:08:36.246462 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.234120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:50635 (local address 127.27.109.1:37181)
0811 03:08:36.234359 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.234379 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.234968 (+ 589us) server_negotiation.cc:407] Connection header received
0811 03:08:36.235181 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.235186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.235266 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.235366 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:36.236310 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.237531 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.243541 (+ 6010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.244895 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.245007 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.245086 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.245908 (+ 822us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.246133 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 03:08:36.247467 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.233952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.27.109.2:50635)
0811 03:08:36.234865 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.234883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:36.235033 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:36.235399 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:36.235404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:36.235444 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:36.236115 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.236130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.237732 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.237736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.240652 (+ 2916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.240676 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.245401 (+ 4725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.245410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.245659 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.245712 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 03:08:36.246737 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0811 03:08:36.246942 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":96,"spinlock_wait_cycles":46720,"thread_start_us":509,"threads_started":1}
I20260811 03:08:36.247771 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.234010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:60001 (local address 127.27.109.3:42219)
0811 03:08:36.234246 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.234255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.234954 (+ 699us) server_negotiation.cc:407] Connection header received
0811 03:08:36.235181 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.235186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.235266 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.235366 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:36.236309 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.237531 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.243496 (+ 5965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.244787 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.244977 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.245058 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.247343 (+ 2285us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.247512 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:08:36.248647 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.233829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:42219 (local address 127.27.109.2:60001)
0811 03:08:36.234783 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.234803 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:36.234982 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:36.235399 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:36.235404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:36.235433 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:36.236116 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.236130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.237731 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.237736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.243314 (+ 5578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.243336 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.246413 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.246420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.246627 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.246666 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 03:08:36.248094 (+ 1428us) client_negotiation.cc:240] Negotiation successful
0811 03:08:36.248249 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":2612,"thread_start_us":567,"threads_started":1}
I20260811 03:08:36.248857 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43236bb0717d4e49b395d28e2956ac3f" candidate_uuid: "60ba0342b9c045f68eed2932060444de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546615372ca340ad8326852c668e837e" is_pre_election: true
I20260811 03:08:36.249522 13322 raft_consensus.cc:2454] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba0342b9c045f68eed2932060444de in term 0.
I20260811 03:08:36.250659 13616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43236bb0717d4e49b395d28e2956ac3f" candidate_uuid: "60ba0342b9c045f68eed2932060444de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7bd27874f4700a0fc11658f040935" is_pre_election: true
I20260811 03:08:36.251448 13616 raft_consensus.cc:2454] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba0342b9c045f68eed2932060444de in term 0.
I20260811 03:08:36.252457 13402 leader_election.cc:302] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd7bd27874f4700a0fc11658f040935, 60ba0342b9c045f68eed2932060444de; no voters:
I20260811 03:08:36.253028 13693 raft_consensus.cc:2781] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:08:36.253212 13693 raft_consensus.cc:482] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:08:36.253324 13693 raft_consensus.cc:3037] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:36.257999 13693 raft_consensus.cc:504] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.260725 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.238776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:57281 (local address 127.27.109.3:42219)
0811 03:08:36.248024 (+ 9248us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.248030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.251326 (+ 3296us) server_negotiation.cc:407] Connection header received
0811 03:08:36.251660 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.251669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.251768 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.251849 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:36.252922 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.254517 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.257638 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.258979 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.259149 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.259231 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.260242 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.260396 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9129,"thread_start_us":21137,"threads_started":1}
I20260811 03:08:36.260736 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.238245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:42219 (local address 127.27.109.1:57281)
0811 03:08:36.251198 (+ 12953us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.251224 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:36.251463 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:36.252158 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:36.252165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:36.252201 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:36.252724 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.252738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.254782 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.254790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.257453 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.257469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.259463 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.259470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.259645 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.259682 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 03:08:36.260175 (+ 493us) client_negotiation.cc:240] Negotiation successful
0811 03:08:36.260337 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12761,"thread_start_us":17117,"threads_started":1}
I20260811 03:08:36.260814 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43236bb0717d4e49b395d28e2956ac3f" candidate_uuid: "60ba0342b9c045f68eed2932060444de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546615372ca340ad8326852c668e837e"
I20260811 03:08:36.261059 13322 raft_consensus.cc:3037] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:36.261797 13693 leader_election.cc:288] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [CANDIDATE]: Term 1 election: Requested vote from peers 546615372ca340ad8326852c668e837e (127.27.109.1:37181), 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219)
I20260811 03:08:36.261744 13616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43236bb0717d4e49b395d28e2956ac3f" candidate_uuid: "546615372ca340ad8326852c668e837e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7bd27874f4700a0fc11658f040935" is_pre_election: true
I20260811 03:08:36.261974 13616 raft_consensus.cc:2454] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 546615372ca340ad8326852c668e837e in term 0.
I20260811 03:08:36.262614 13252 leader_election.cc:302] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd7bd27874f4700a0fc11658f040935, 546615372ca340ad8326852c668e837e; no voters:
I20260811 03:08:36.263038 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.239714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:50569 (local address 127.27.109.2:43893)
0811 03:08:36.239928 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.239936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.255351 (+ 15415us) server_negotiation.cc:407] Connection header received
0811 03:08:36.255830 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.255836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.255924 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.255994 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:36.256895 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.257956 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.260554 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.261896 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.262050 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.262140 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.262614 (+ 474us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.262785 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 03:08:36.263132 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.238997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43893 (local address 127.27.109.1:50569)
0811 03:08:36.255247 (+ 16250us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.255267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:36.255644 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:36.256188 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:36.256195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:36.256220 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:36.256694 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.256708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.258143 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.258148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.260348 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:36.260362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.262228 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.262233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:36.262412 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.262449 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 03:08:36.262554 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 03:08:36.262754 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16099,"thread_start_us":11657,"threads_started":1}
I20260811 03:08:36.264005 13616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43236bb0717d4e49b395d28e2956ac3f" candidate_uuid: "60ba0342b9c045f68eed2932060444de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7bd27874f4700a0fc11658f040935"
I20260811 03:08:36.264228 13616 raft_consensus.cc:3037] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:36.264191 13471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43236bb0717d4e49b395d28e2956ac3f" candidate_uuid: "546615372ca340ad8326852c668e837e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba0342b9c045f68eed2932060444de" is_pre_election: true
I20260811 03:08:36.264722 13322 raft_consensus.cc:2454] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba0342b9c045f68eed2932060444de in term 1.
I20260811 03:08:36.264793 13471 raft_consensus.cc:2379] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 546615372ca340ad8326852c668e837e in current term 1: Already voted for candidate 60ba0342b9c045f68eed2932060444de in this term.
I20260811 03:08:36.265257 13400 leader_election.cc:302] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546615372ca340ad8326852c668e837e, 60ba0342b9c045f68eed2932060444de; no voters:
I20260811 03:08:36.265622 13693 raft_consensus.cc:2781] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:08:36.265909 13691 raft_consensus.cc:2741] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260811 03:08:36.266566 13693 raft_consensus.cc:686] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 1 LEADER]: Becoming Leader. State: Replica: 60ba0342b9c045f68eed2932060444de, State: Running, Role: LEADER
I20260811 03:08:36.267221 13693 consensus_queue.cc:237] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } }
I20260811 03:08:36.267868 13616 raft_consensus.cc:2454] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba0342b9c045f68eed2932060444de in term 1.
I20260811 03:08:36.272104 13165 catalog_manager.cc:5495] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba0342b9c045f68eed2932060444de (127.27.109.2). New cstate: current_term: 1 leader_uuid: "60ba0342b9c045f68eed2932060444de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } health_report { overall_health: HEALTHY } } }
W20260811 03:08:36.313446 13665 tablet.cc:2369] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:36.315828 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.309686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.27.109.2:43893)
0811 03:08:36.309856 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.309861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.309900 (+ 39us) server_negotiation.cc:407] Connection header received
0811 03:08:36.310171 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.310175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.310256 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.310390 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:08:36.310884 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.311696 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.313503 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.313955 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.314068 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.314728 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:08:36.315263 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:08:36.315384 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.315504 (+ 120us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.315589 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 03:08:36.324913 13322 raft_consensus.cc:1261] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Refusing update from remote peer 60ba0342b9c045f68eed2932060444de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:36.324944 13616 raft_consensus.cc:1261] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 1 FOLLOWER]: Refusing update from remote peer 60ba0342b9c045f68eed2932060444de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:36.325989 13699 consensus_queue.cc:1035] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:36.326375 13693 consensus_queue.cc:1035] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [LEADER]: Connected to new peer: Peer: permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:36.356539 13165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:08:36.356882 13165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:08:36.361258 13165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Sending DeleteTablet for 3 replicas of tablet 43236bb0717d4e49b395d28e2956ac3f
I20260811 03:08:36.362712 13301 tablet_service.cc:1514] Processing DeleteTablet for tablet 43236bb0717d4e49b395d28e2956ac3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:36 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:37920
I20260811 03:08:36.362685 13592 tablet_service.cc:1514] Processing DeleteTablet for tablet 43236bb0717d4e49b395d28e2956ac3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:36 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:34224
I20260811 03:08:36.363104 13449 tablet_service.cc:1514] Processing DeleteTablet for tablet 43236bb0717d4e49b395d28e2956ac3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:36 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:44644
I20260811 03:08:36.363736 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 13232
I20260811 03:08:36.394086 13715 tablet_replica.cc:330] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: stopping tablet replica
I20260811 03:08:36.394253 13714 tablet_replica.cc:330] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: stopping tablet replica
I20260811 03:08:36.395046 13714 raft_consensus.cc:2229] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:08:36.395721 13714 raft_consensus.cc:2258] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:36.396843 13716 tablet_replica.cc:330] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: stopping tablet replica
I20260811 03:08:36.397174 13715 raft_consensus.cc:2229] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:08:36.397770 13715 raft_consensus.cc:2258] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:36.398581 13716 raft_consensus.cc:2229] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:08:36.398905 13714 ts_tablet_manager.cc:1905] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:36.399377 13716 raft_consensus.cc:2258] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:36.401823 13716 ts_tablet_manager.cc:1905] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:36.403381 13715 ts_tablet_manager.cc:1905] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 03:08:36 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:08:36.407117 13714 ts_tablet_manager.cc:1918] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:36.407435 13714 log.cc:1196] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/43236bb0717d4e49b395d28e2956ac3f
I20260811 03:08:36.408133 13714 ts_tablet_manager.cc:1939] T 43236bb0717d4e49b395d28e2956ac3f P 1fd7bd27874f4700a0fc11658f040935: Deleting consensus metadata
I20260811 03:08:36.410421 13152 catalog_manager.cc:4841] TS 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219): tablet 43236bb0717d4e49b395d28e2956ac3f (table test-table [id=4e0ee1d07b6d47618adf4646f436821c]) successfully deleted
I20260811 03:08:36.413260 13716 ts_tablet_manager.cc:1918] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:36.413465 13716 log.cc:1196] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/43236bb0717d4e49b395d28e2956ac3f
I20260811 03:08:36.414103 13716 ts_tablet_manager.cc:1939] T 43236bb0717d4e49b395d28e2956ac3f P 60ba0342b9c045f68eed2932060444de: Deleting consensus metadata
I20260811 03:08:36.414306 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.365218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.27.109.1:37181)
0811 03:08:36.365458 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.365479 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.388054 (+ 22575us) server_negotiation.cc:407] Connection header received
0811 03:08:36.388793 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.388803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.388940 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.389191 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:36.394532 (+ 5341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.396336 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.398485 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.399153 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.399258 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.406029 (+ 6771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:36.406046 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:36.406066 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:36.406117 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:36.410762 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.411743 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.411754 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.411758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.411974 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.412416 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.412422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.412425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.412906 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:36.413240 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.413753 (+ 513us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.413975 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:08:36.415742 13367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:36.419636 13151 catalog_manager.cc:4841] TS 60ba0342b9c045f68eed2932060444de (127.27.109.2:43893): tablet 43236bb0717d4e49b395d28e2956ac3f (table test-table [id=4e0ee1d07b6d47618adf4646f436821c]) successfully deleted
I20260811 03:08:36.524976 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 13232
W20260811 03:08:36.548322 13150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:37181 (error 108)
I20260811 03:08:36.549165 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 13382
I20260811 03:08:36.550771 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.550146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:37950)
0811 03:08:36.550431 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.550555 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260811 03:08:36.551198 13150 catalog_manager.cc:4584] TS 546615372ca340ad8326852c668e837e (127.27.109.1:37181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43236bb0717d4e49b395d28e2956ac3f: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Aug 11 03:08:36 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:08:36.567585 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.549983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.27.109.2:43893)
0811 03:08:36.550180 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.550187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.550324 (+ 137us) server_negotiation.cc:407] Connection header received
0811 03:08:36.550569 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.550578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.550663 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.550841 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:36.552304 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.553074 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.554520 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.555202 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.555358 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.560455 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:36.560469 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:36.560485 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:36.560543 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:36.564179 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.565139 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.565147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.565149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.565305 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.565726 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.565731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.565745 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.566195 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:36.566541 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.566955 (+ 414us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.567231 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 03:08:36.568400 13516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:36.604954 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.604323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:37956)
0811 03:08:36.604599 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.604732 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260811 03:08:36.606148 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.605731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:37966)
0811 03:08:36.605894 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.605999 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 03:08:36.667120 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.666478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:37976)
0811 03:08:36.666786 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.666939 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260811 03:08:36.668306 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.667923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:37982)
0811 03:08:36.668089 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.668182 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 03:08:36.692656 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 13382
I20260811 03:08:36.718736 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 13527
Aug 11 03:08:36 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:08:36.735795 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.719575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34240 (local address 127.27.109.3:42219)
0811 03:08:36.719767 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:36.719791 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:36.719822 (+ 31us) server_negotiation.cc:407] Connection header received
0811 03:08:36.719916 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:36.719924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:36.720020 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:36.720152 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:36.721963 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.722725 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.724198 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:36.724623 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:36.724725 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:36.729717 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:36.729735 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:36.729754 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:36.729803 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:36.732608 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.733385 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.733395 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.733398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.733556 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:36.733990 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:36.733997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:36.733999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:36.734522 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:36.734854 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:36.735187 (+ 333us) server_negotiation.cc:299] Negotiation successful
0811 03:08:36.735402 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 03:08:36.736651 13659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:36.749341 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.748626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:37990)
0811 03:08:36.748925 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.749065 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260811 03:08:36.750808 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.750228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:38000)
0811 03:08:36.750478 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.750601 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260811 03:08:36.831511 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 13527
I20260811 03:08:36.855064 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 13131
I20260811 03:08:36.856441 13198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:36.973484 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 13131
I20260811 03:08:36.973575 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:36.972786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:38016)
0811 03:08:36.973161 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:36.973292 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260811 03:08:57.003521 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41965
--webserver_interface=127.27.109.62
--webserver_port=44181
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:08:57.380082 13723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:57.380411 13723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:57.380600 13723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:57.388459 13723 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 03:08:57.388578 13723 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 03:08:57.388646 13723 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 03:08:57.388725 13723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:08:57.388823 13723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:57.388922 13723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:57.388965 13723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:08:57.389014 13723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:08:57.400032 13723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=44181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:57.401520 13723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:57.403144 13723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:57.432061 13730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:57.432112 13732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:57.432019 13734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:57.437142 13723 server_base.cc:1034] running on GCE node
I20260811 03:08:57.438243 13723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:57.440237 13723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:57.441606 13723 hybrid_clock.cc:648] HybridClock initialized: now 1786417737441522 us; error 42 us; skew 500 ppm
I20260811 03:08:57.442225 13723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:57 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417737, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:57.446769 13723 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:08:57.448930 13723 webserver.cc:460] Webserver started at http://127.27.109.62:44181/ using document root <none> and password file <none>
I20260811 03:08:57.449891 13723 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:57.450014 13723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:57.456828 13723 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260811 03:08:57.460309 13741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:57.461982 13723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:08:57.462219 13723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "3352b7df0d7f4338bb37b5ca7a28defe"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "4fdbfe19c22ded3d64e4a060efa53185"
server_key_iv: "3847231c39e8ccb8767d7ceef01cb62f"
server_key_version: "encryptionkey@0"
I20260811 03:08:57.463493 13723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:57.477807 13723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:57.483040 13723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:57.483443 13723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:57.504873 13723 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41965
I20260811 03:08:57.504894 13795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41965 every 8 connection(s)
I20260811 03:08:57.506660 13723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:08:57.511194 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 13723
I20260811 03:08:57.512084 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:37181
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:41965
--webserver_port=36937
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:08:57.515722 13796 sys_catalog.cc:263] Verifying existing consensus state
I20260811 03:08:57.519850 13796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Bootstrap starting.
I20260811 03:08:57.543545 13796 log.cc:824] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:57.558279 13796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 03:08:57.559087 13796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Bootstrap complete.
I20260811 03:08:57.570639 13796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:08:57.571941 13796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3352b7df0d7f4338bb37b5ca7a28defe, State: Initialized, Role: FOLLOWER
I20260811 03:08:57.572817 13796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:08:57.573148 13796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:08:57.573320 13796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:08:57.573549 13796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 1 FOLLOWER]: Advancing to term 2
I20260811 03:08:57.578485 13796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:08:57.579264 13796 leader_election.cc:302] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3352b7df0d7f4338bb37b5ca7a28defe; no voters:
I20260811 03:08:57.580081 13796 leader_election.cc:288] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 03:08:57.580251 13802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 2 FOLLOWER]: Leader election won for term 2
I20260811 03:08:57.581625 13802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 2 LEADER]: Becoming Leader. State: Replica: 3352b7df0d7f4338bb37b5ca7a28defe, State: Running, Role: LEADER
I20260811 03:08:57.582341 13802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:08:57.583060 13796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:08:57.585196 13804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3352b7df0d7f4338bb37b5ca7a28defe. Latest consensus state: current_term: 2 leader_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } } }
I20260811 03:08:57.585680 13804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:57.586163 13803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } } }
I20260811 03:08:57.586488 13803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: This master's current role is: LEADER
I20260811 03:08:57.590843 13807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:08:57.603451 13807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:08:57.605744 13807 catalog_manager.cc:1259] Loaded cluster ID: cbdf9e92667f4fd9979214722d56aef4
I20260811 03:08:57.606016 13807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:08:57.613231 13807 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:08:57.615078 13807 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Loaded TSK: 0
I20260811 03:08:57.636989 13807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Generated new TSK 1
I20260811 03:08:57.637593 13807 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 03:08:57.924613 13798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:57.924919 13798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:57.925078 13798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:57.932628 13798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:57.932755 13798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:57.932961 13798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:08:57.943665 13798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:37181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=36937
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:57.945187 13798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:57.946595 13798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:57.962982 13828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:57.963227 13827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:57.963045 13830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:57.964355 13798 server_base.cc:1034] running on GCE node
I20260811 03:08:57.965116 13798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:57.966207 13798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:57.967523 13798 hybrid_clock.cc:648] HybridClock initialized: now 1786417737967441 us; error 54 us; skew 500 ppm
I20260811 03:08:57.967924 13798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:57 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417737, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:57.971572 13798 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:08:57.973285 13798 webserver.cc:460] Webserver started at http://127.27.109.1:36937/ using document root <none> and password file <none>
I20260811 03:08:57.974041 13798 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:57.974195 13798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:57.980511 13798 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260811 03:08:57.984170 13838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:57.986002 13798 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 03:08:57.986212 13798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "546615372ca340ad8326852c668e837e"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "9e0540de41198c17bc45fc8989c32a3a"
server_key_iv: "367a8104a18f7fae04b296c93d9810fd"
server_key_version: "encryptionkey@0"
I20260811 03:08:57.987385 13798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:58.055795 13798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:58.060369 13798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:58.060690 13798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:58.061761 13798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:58.063838 13846 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260811 03:08:58.070062 13846 ts_tablet_manager.cc:1852] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260811 03:08:58.071158 13846 ts_tablet_manager.cc:1905] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:58.078900 13846 ts_tablet_manager.cc:1918] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:58.079165 13846 log.cc:1196] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/43236bb0717d4e49b395d28e2956ac3f
I20260811 03:08:58.079722 13846 ts_tablet_manager.cc:1939] T 43236bb0717d4e49b395d28e2956ac3f P 546615372ca340ad8326852c668e837e: Deleting consensus metadata
I20260811 03:08:58.080170 13798 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260811 03:08:58.080420 13798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.001s
I20260811 03:08:58.080585 13798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:58.080683 13798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:58.118290 13798 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37181
I20260811 03:08:58.118330 13962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37181 every 8 connection(s)
I20260811 03:08:58.119848 13798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:08:58.128661 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 13798
I20260811 03:08:58.129369 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:43893
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:41965
--webserver_port=41917
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:08:58 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417737, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:58.146863 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:58.122902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:45055 (local address 127.27.109.62:41965)
0811 03:08:58.123586 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:58.123608 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:58.124194 (+ 586us) server_negotiation.cc:407] Connection header received
0811 03:08:58.126363 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:58.126377 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:58.126794 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:58.127103 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:58.128675 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:58.130040 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:58.131826 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:58.132481 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:58.132692 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:58.136938 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:58.136952 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:58.136978 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:58.137045 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:58.141554 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:58.142831 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:58.142839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:58.142844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:58.143011 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:58.143585 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:58.143588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:58.143590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:58.144178 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:58.144340 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:58.145382 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0811 03:08:58.145749 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":249,"threads_started":1}
I20260811 03:08:58.147349 13966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:58.123152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:45055)
0811 03:08:58.124010 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:58.124054 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:58.125809 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:58.127265 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:58.127273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:58.127822 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:58.128453 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:58.128472 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:58.130231 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:58.130235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:58.131663 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:58.131677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:58.132753 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:58.132761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:58.133059 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:58.133787 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:58.133810 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:58.136619 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:58.141800 (+ 5181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:58.141811 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:58.141832 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:58.142642 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:58.143151 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:58.143157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:58.143162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:58.143433 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:58.144328 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:58.144339 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:58.144836 (+ 497us) client_negotiation.cc:769] Sending connection context
0811 03:08:58.145176 (+ 340us) client_negotiation.cc:240] Negotiation successful
0811 03:08:58.145822 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":336,"threads_started":1}
I20260811 03:08:58.150274 13963 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41965
I20260811 03:08:58.150642 13963 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:58.151430 13963 heartbeater.cc:511] Master 127.27.109.62:41965 requested a full tablet report, sending...
I20260811 03:08:58.153724 13758 ts_manager.cc:194] Registered new tserver with Master: 546615372ca340ad8326852c668e837e (127.27.109.1:37181)
I20260811 03:08:58.155521 13758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:45055
W20260811 03:08:58.500833 13967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:58.501143 13967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:58.501298 13967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:58.509163 13967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:58.509367 13967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:58.509567 13967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:08:58.520263 13967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:43893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=41917
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:58.521967 13967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:58.523541 13967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:58.538357 13972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:58.538302 13973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:58.538271 13976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:58.539175 13967 server_base.cc:1034] running on GCE node
I20260811 03:08:58.539937 13967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:58.541074 13967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:58.542322 13967 hybrid_clock.cc:648] HybridClock initialized: now 1786417738542264 us; error 74 us; skew 500 ppm
I20260811 03:08:58.542764 13967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:58 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417738, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:58.546279 13967 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:08:58.547994 13967 webserver.cc:460] Webserver started at http://127.27.109.2:41917/ using document root <none> and password file <none>
I20260811 03:08:58.548806 13967 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:58.548987 13967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:58.555166 13967 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:08:58.558321 13983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:58.559844 13967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:08:58.560053 13967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "60ba0342b9c045f68eed2932060444de"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "3b2632bb99eafcff8f03fcda983257b2"
server_key_iv: "ef9c57ed3d88be5396a33e48d5b22385"
server_key_version: "encryptionkey@0"
I20260811 03:08:58.561189 13967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:58.623926 13967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:58.628638 13967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:58.628939 13967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:58.630007 13967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:58.631479 13967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:58.631608 13967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:58.631754 13967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:58.631846 13967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:58.669185 13967 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43893
I20260811 03:08:58.669288 14106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43893 every 8 connection(s)
I20260811 03:08:58.670976 13967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:08:58.677331 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 13967
I20260811 03:08:58.678047 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:42219
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:41965
--webserver_port=40835
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 03:08:58 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417738, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:58.696062 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:58.673712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:58351 (local address 127.27.109.62:41965)
0811 03:08:58.674237 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:58.674242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:58.675309 (+ 1067us) server_negotiation.cc:407] Connection header received
0811 03:08:58.677254 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:58.677259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:58.677356 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:58.677493 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:58.679318 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:58.680060 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:58.681689 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:58.682109 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:58.682258 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:58.687304 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:58.687314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:58.687323 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:58.687386 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:58.691373 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:58.692734 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:58.692740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:58.692746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:58.692946 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:58.693427 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:58.693435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:58.693441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:58.693847 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:58.694021 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:58.695177 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0811 03:08:58.695505 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":305,"threads_started":1}
I20260811 03:08:58.696775 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:58.674241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:58351)
0811 03:08:58.675121 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:58.675172 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:58.676931 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:58.677749 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:58.677759 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:58.678422 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:58.679118 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:58.679138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:58.680251 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:58.680255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:58.681525 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:58.681538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:58.682392 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:58.682400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:58.682648 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:58.683648 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:58.683709 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:58.686921 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:58.691581 (+ 4660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:58.691595 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:58.691626 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:58.692544 (+ 918us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:58.693083 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:58.693087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:58.693089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:58.693291 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:58.694212 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:58.694231 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:58.694628 (+ 397us) client_negotiation.cc:769] Sending connection context
0811 03:08:58.695013 (+ 385us) client_negotiation.cc:240] Negotiation successful
0811 03:08:58.695484 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":279,"threads_started":1}
I20260811 03:08:58.698660 14107 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41965
I20260811 03:08:58.699147 14107 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:58.700126 14107 heartbeater.cc:511] Master 127.27.109.62:41965 requested a full tablet report, sending...
I20260811 03:08:58.702283 13758 ts_manager.cc:194] Registered new tserver with Master: 60ba0342b9c045f68eed2932060444de (127.27.109.2:43893)
I20260811 03:08:58.703266 13758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:58351
W20260811 03:08:59.060305 14111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:08:59.060639 14111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:08:59.060808 14111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:08:59.069156 14111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:08:59.069294 14111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:08:59.069471 14111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:08:59.080130 14111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:42219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=40835
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:08:59.081573 14111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:08:59.083173 14111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:08:59.098507 14119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:59.098538 14116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:08:59.098588 14117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:08:59.099424 14111 server_base.cc:1034] running on GCE node
I20260811 03:08:59.100163 14111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:08:59.101277 14111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:08:59.102556 14111 hybrid_clock.cc:648] HybridClock initialized: now 1786417739102494 us; error 55 us; skew 500 ppm
I20260811 03:08:59.102934 14111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:08:59 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417739, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:08:59.106428 14111 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:08:59.108225 14111 webserver.cc:460] Webserver started at http://127.27.109.3:40835/ using document root <none> and password file <none>
I20260811 03:08:59.108989 14111 fs_manager.cc:359] Metadata directory not provided
I20260811 03:08:59.109165 14111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:08:59.115233 14111 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:08:59.118459 14126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:59.120407 14111 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:08:59.120627 14111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "1fd7bd27874f4700a0fc11658f040935"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "411f7a672f57e8f870222fe5512daa96"
server_key_iv: "7af77c0c4995df10feaa3497057b57bc"
server_key_version: "encryptionkey@0"
I20260811 03:08:59.122038 14111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:08:59.137641 14111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:08:59.142495 14111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:08:59.142853 14111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:08:59.143959 14111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:08:59.145491 14111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:08:59.145584 14111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:59.145663 14111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:08:59.145718 14111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:08:59.158996 13963 heartbeater.cc:503] Master 127.27.109.62:41965 was elected leader, sending a full tablet report...
I20260811 03:08:59.182603 14111 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:42219
I20260811 03:08:59.182788 14248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:42219 every 8 connection(s)
I20260811 03:08:59.184404 14111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:08:59.188021 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 14111
Aug 11 03:08:59 dist-test-slave-8tdl krb5kdc[13115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417739, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:08:59.206868 14253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.189250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.27.109.62:41965)
0811 03:08:59.190398 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.190403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.190419 (+ 16us) server_negotiation.cc:407] Connection header received
0811 03:08:59.190475 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.190479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.190541 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.190676 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:59.193029 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.193911 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.195659 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.196144 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.196298 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.200007 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:59.200014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:59.200018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:59.200051 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:59.203593 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.204644 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.204648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.204650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.204788 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.205269 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.205273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.205275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.205579 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:59.205980 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.206155 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.206398 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":789,"threads_started":1}
I20260811 03:08:59.209084 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.187553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:49017 (local address 127.27.109.62:41965)
0811 03:08:59.187816 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.187824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.189181 (+ 1357us) server_negotiation.cc:407] Connection header received
0811 03:08:59.191891 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.191897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.192002 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.192122 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:59.193798 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.194594 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.196237 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.196686 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.196823 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.201027 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:59.201033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:59.201036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:59.201080 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:59.204732 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.205668 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.205674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.205678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.206033 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.206593 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.206600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.206607 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.207026 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:59.207193 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.208245 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.208590 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 03:08:59.210311 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.188052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:49017)
0811 03:08:59.188997 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.189056 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:59.191497 (+ 2441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:59.192325 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:59.192334 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:59.192979 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:08:59.193624 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.193643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.194801 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.194805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.196086 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.196099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.196941 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.196949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.197211 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.197918 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:08:59.197942 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:08:59.200752 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:08:59.204920 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:59.204931 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:59.204960 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:59.205521 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:59.206197 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:08:59.206203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:08:59.206206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:08:59.206449 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:08:59.207226 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:08:59.207234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:08:59.207744 (+ 510us) client_negotiation.cc:769] Sending connection context
0811 03:08:59.208139 (+ 395us) client_negotiation.cc:240] Negotiation successful
0811 03:08:59.208741 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":343,"threads_started":1}
I20260811 03:08:59.212074 14249 heartbeater.cc:347] Connected to a master server at 127.27.109.62:41965
I20260811 03:08:59.212476 14249 heartbeater.cc:464] Registering TS with master...
I20260811 03:08:59.213550 14249 heartbeater.cc:511] Master 127.27.109.62:41965 requested a full tablet report, sending...
I20260811 03:08:59.215677 13758 ts_manager.cc:194] Registered new tserver with Master: 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219)
I20260811 03:08:59.216687 13758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:49017
I20260811 03:08:59.223204 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:08:59.230830 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.224245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.27.109.62:41965)
0811 03:08:59.224494 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.224501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.224957 (+ 456us) server_negotiation.cc:407] Connection header received
0811 03:08:59.225086 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.225090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.225262 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.225446 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:08:59.225871 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.226953 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.228666 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.229177 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.229293 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.229848 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:08:59.230149 (+ 301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0811 03:08:59.230496 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37070: token expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260811 03:08:59.230967 14109 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37070: token expired
W20260811 03:08:59.231561 13671 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.109.62:41965)
I20260811 03:08:59.246590 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.232106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.27.109.62:41965)
0811 03:08:59.232279 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.232283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.232299 (+ 16us) server_negotiation.cc:407] Connection header received
0811 03:08:59.232418 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.232421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.232484 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.232569 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:59.234226 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.235071 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.236697 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.237158 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.237259 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.240516 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:59.240521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:59.240524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:59.240568 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:59.243371 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.244165 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.244169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.244172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.244309 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.244845 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.244848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.244850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.245281 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:59.245496 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.245918 (+ 422us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.246310 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 03:08:59.251168 13758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:08:59.258944 13758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:08:59.300861 14263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.272781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:43893 (local address 127.0.0.1:57348)
0811 03:08:59.289698 (+ 16917us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.289778 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:59.290087 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:59.291708 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:59.291720 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:59.291760 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:59.292899 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.292924 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.294976 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.294981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.297414 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.297426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.299886 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.299893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.300110 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.300174 (+ 64us) client_negotiation.cc:769] Sending connection context
0811 03:08:59.300324 (+ 150us) client_negotiation.cc:240] Negotiation successful
0811 03:08:59.300518 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16517,"spinlock_wait_cycles":23808,"thread_start_us":20420,"threads_started":1}
I20260811 03:08:59.302008 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.273004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.27.109.2:43893)
0811 03:08:59.288500 (+ 15496us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.288516 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.290392 (+ 1876us) server_negotiation.cc:407] Connection header received
0811 03:08:59.290548 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.290559 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.290971 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.291315 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:59.293197 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.294780 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.297582 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.299300 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.299452 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.299687 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.301423 (+ 1736us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.301591 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15208,"thread_start_us":15555,"threads_started":1}
I20260811 03:08:59.303539 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.275306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39268)
0811 03:08:59.293606 (+ 18300us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.293632 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:59.293800 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:59.294451 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:59.294456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:59.294488 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:59.294942 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.294958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.296943 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.296951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.299492 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.299508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.301932 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.301938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.302817 (+ 879us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.302864 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 03:08:59.302976 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 03:08:59.303150 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18097,"thread_start_us":13766,"threads_started":1}
I20260811 03:08:59.303620 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.275623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.27.109.1:37181)
0811 03:08:59.292710 (+ 17087us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.292723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.293822 (+ 1099us) server_negotiation.cc:407] Connection header received
0811 03:08:59.293943 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.293951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.294213 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.294400 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:59.295198 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.296772 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.299731 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.301708 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.301860 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.302211 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.303081 (+ 870us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.303257 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16881,"thread_start_us":16767,"threads_started":1}
I20260811 03:08:59.304386 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.273357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:42219 (local address 127.0.0.1:60188)
0811 03:08:59.291666 (+ 18309us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.291694 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:59.291872 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:59.295471 (+ 3599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:59.295475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:59.295509 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:59.296199 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.296215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.299021 (+ 2806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.299027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.301618 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.301637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.303700 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.303711 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.303948 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.303975 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 03:08:59.304034 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 03:08:59.304133 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18111,"thread_start_us":18706,"threads_started":1}
I20260811 03:08:59.305743 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.273441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.27.109.3:42219)
0811 03:08:59.294486 (+ 21045us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.294496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.294532 (+ 36us) server_negotiation.cc:407] Connection header received
0811 03:08:59.294639 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.294646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.294987 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.296418 (+ 1431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:59.296755 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.298737 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.301887 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.303395 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.304702 (+ 1307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.304977 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.305062 (+ 85us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.305250 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20866,"thread_start_us":20671,"threads_started":1}
I20260811 03:08:59.305853 13894 tablet_service.cc:1467] Processing CreateTablet for tablet 8faf8cfc261d41c08cd04548ca151314 (DEFAULT_TABLE table=test-table [id=77f16179803b46039b4e6e0fa271eec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:59.306116 14039 tablet_service.cc:1467] Processing CreateTablet for tablet 8faf8cfc261d41c08cd04548ca151314 (DEFAULT_TABLE table=test-table [id=77f16179803b46039b4e6e0fa271eec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:59.307646 13894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8faf8cfc261d41c08cd04548ca151314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:59.308027 14039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8faf8cfc261d41c08cd04548ca151314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:59.310084 14180 tablet_service.cc:1467] Processing CreateTablet for tablet 8faf8cfc261d41c08cd04548ca151314 (DEFAULT_TABLE table=test-table [id=77f16179803b46039b4e6e0fa271eec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:08:59.311668 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8faf8cfc261d41c08cd04548ca151314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:08:59.318528 14265 tablet_bootstrap.cc:492] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Bootstrap starting.
I20260811 03:08:59.322796 14265 tablet_bootstrap.cc:654] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:59.322935 14266 tablet_bootstrap.cc:492] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Bootstrap starting.
I20260811 03:08:59.324421 14265 log.cc:824] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:59.327023 14266 tablet_bootstrap.cc:654] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:59.327862 14265 tablet_bootstrap.cc:492] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: No bootstrap required, opened a new log
I20260811 03:08:59.328166 14265 ts_tablet_manager.cc:1397] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260811 03:08:59.328751 14266 log.cc:824] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:59.331785 14268 tablet_bootstrap.cc:492] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Bootstrap starting.
I20260811 03:08:59.333808 14266 tablet_bootstrap.cc:492] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: No bootstrap required, opened a new log
I20260811 03:08:59.334129 14266 ts_tablet_manager.cc:1397] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260811 03:08:59.335304 14268 tablet_bootstrap.cc:654] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Neither blocks nor log segments found. Creating new log.
I20260811 03:08:59.336565 14268 log.cc:824] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Log is configured to *not* fsync() on all Append() calls
I20260811 03:08:59.340109 14268 tablet_bootstrap.cc:492] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: No bootstrap required, opened a new log
I20260811 03:08:59.339946 14265 raft_consensus.cc:348] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.340436 14268 ts_tablet_manager.cc:1397] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260811 03:08:59.340432 14265 raft_consensus.cc:374] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:59.340528 14265 raft_consensus.cc:729] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546615372ca340ad8326852c668e837e, State: Initialized, Role: FOLLOWER
I20260811 03:08:59.341228 14265 consensus_queue.cc:260] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.343183 14265 ts_tablet_manager.cc:1428] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260811 03:08:59.344852 14266 raft_consensus.cc:348] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.345378 14266 raft_consensus.cc:374] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:59.345500 14266 raft_consensus.cc:729] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba0342b9c045f68eed2932060444de, State: Initialized, Role: FOLLOWER
I20260811 03:08:59.346136 14266 consensus_queue.cc:260] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.347481 14107 heartbeater.cc:503] Master 127.27.109.62:41965 was elected leader, sending a full tablet report...
I20260811 03:08:59.348131 14266 ts_tablet_manager.cc:1428] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260811 03:08:59.353121 14268 raft_consensus.cc:348] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.353680 14268 raft_consensus.cc:374] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:08:59.353814 14268 raft_consensus.cc:729] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd7bd27874f4700a0fc11658f040935, State: Initialized, Role: FOLLOWER
I20260811 03:08:59.354424 14268 consensus_queue.cc:260] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.355654 14249 heartbeater.cc:503] Master 127.27.109.62:41965 was elected leader, sending a full tablet report...
I20260811 03:08:59.356356 14268 ts_tablet_manager.cc:1428] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260811 03:08:59.358129 14272 raft_consensus.cc:482] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:08:59.358429 14272 raft_consensus.cc:504] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.360285 14272 leader_election.cc:288] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219), 546615372ca340ad8326852c668e837e (127.27.109.1:37181)
I20260811 03:08:59.368510 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.360973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:34709 (local address 127.27.109.3:42219)
0811 03:08:59.361180 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.361189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.361634 (+ 445us) server_negotiation.cc:407] Connection header received
0811 03:08:59.361931 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.361934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.361992 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.362051 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:59.362900 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.364189 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.366445 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.367371 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.367575 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.367656 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.368035 (+ 379us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.368155 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 03:08:59.368538 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.360517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:42219 (local address 127.27.109.2:34709)
0811 03:08:59.361525 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.361551 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:59.361756 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:59.362211 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:59.362214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:59.362234 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:59.362670 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.362724 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.364440 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.364444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.366271 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.366286 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.367598 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.367604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.367765 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.367814 (+ 49us) client_negotiation.cc:769] Sending connection context
0811 03:08:59.367946 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 03:08:59.368108 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":1195,"threads_started":1}
I20260811 03:08:59.368866 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.360880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.27.109.2:32895)
0811 03:08:59.361716 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.361731 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 03:08:59.361890 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:08:59.362228 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:08:59.362233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:08:59.362257 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:08:59.362766 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.362777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.364438 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.364443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.366283 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:08:59.366297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.367978 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.367984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:08:59.368169 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.368200 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 03:08:59.368300 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 03:08:59.368421 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":1,"thread_start_us":499,"threads_started":1}
I20260811 03:08:59.368933 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.361311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:32895 (local address 127.27.109.1:37181)
0811 03:08:59.361528 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.361535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.361825 (+ 290us) server_negotiation.cc:407] Connection header received
0811 03:08:59.362033 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.362039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.362112 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.362211 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:08:59.362954 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.364252 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.366751 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.367704 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.367908 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.367995 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.368344 (+ 349us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.368507 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 03:08:59.369956 13916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8faf8cfc261d41c08cd04548ca151314" candidate_uuid: "60ba0342b9c045f68eed2932060444de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546615372ca340ad8326852c668e837e" is_pre_election: true
I20260811 03:08:59.370064 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8faf8cfc261d41c08cd04548ca151314" candidate_uuid: "60ba0342b9c045f68eed2932060444de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7bd27874f4700a0fc11658f040935" is_pre_election: true
I20260811 03:08:59.370527 13916 raft_consensus.cc:2454] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba0342b9c045f68eed2932060444de in term 0.
I20260811 03:08:59.370594 14202 raft_consensus.cc:2454] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ba0342b9c045f68eed2932060444de in term 0.
I20260811 03:08:59.371294 13987 leader_election.cc:302] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd7bd27874f4700a0fc11658f040935, 60ba0342b9c045f68eed2932060444de; no voters:
I20260811 03:08:59.371821 14272 raft_consensus.cc:2781] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:08:59.371999 14272 raft_consensus.cc:482] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:08:59.372152 14272 raft_consensus.cc:3037] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 0 FOLLOWER]: Advancing to term 1
W20260811 03:08:59.372380 13964 tablet.cc:2369] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:59.374862 14272 raft_consensus.cc:504] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.375978 14272 leader_election.cc:288] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [CANDIDATE]: Term 1 election: Requested vote from peers 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219), 546615372ca340ad8326852c668e837e (127.27.109.1:37181)
I20260811 03:08:59.376667 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8faf8cfc261d41c08cd04548ca151314" candidate_uuid: "60ba0342b9c045f68eed2932060444de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fd7bd27874f4700a0fc11658f040935"
I20260811 03:08:59.376691 13916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8faf8cfc261d41c08cd04548ca151314" candidate_uuid: "60ba0342b9c045f68eed2932060444de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "546615372ca340ad8326852c668e837e"
I20260811 03:08:59.376945 13916 raft_consensus.cc:3037] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:59.376956 14202 raft_consensus.cc:3037] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:08:59.380750 13916 raft_consensus.cc:2454] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba0342b9c045f68eed2932060444de in term 1.
I20260811 03:08:59.380750 14202 raft_consensus.cc:2454] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ba0342b9c045f68eed2932060444de in term 1.
I20260811 03:08:59.381299 13985 leader_election.cc:302] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 546615372ca340ad8326852c668e837e, 60ba0342b9c045f68eed2932060444de; no voters:
I20260811 03:08:59.381721 14272 raft_consensus.cc:2781] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:08:59.382257 14272 raft_consensus.cc:686] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 1 LEADER]: Becoming Leader. State: Replica: 60ba0342b9c045f68eed2932060444de, State: Running, Role: LEADER
I20260811 03:08:59.382907 14272 consensus_queue.cc:237] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } }
I20260811 03:08:59.388459 13757 catalog_manager.cc:5495] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ba0342b9c045f68eed2932060444de (127.27.109.2). New cstate: current_term: 1 leader_uuid: "60ba0342b9c045f68eed2932060444de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ba0342b9c045f68eed2932060444de" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 43893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 } health_report { overall_health: UNKNOWN } } }
I20260811 03:08:59.413082 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.407199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.27.109.2:43893)
0811 03:08:59.407384 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.407389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.407440 (+ 51us) server_negotiation.cc:407] Connection header received
0811 03:08:59.407702 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.407706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.407805 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.407948 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:08:59.408397 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.409174 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.410618 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.411192 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.411312 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.411863 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:08:59.412463 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:08:59.412588 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.412662 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.412738 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 03:08:59.422228 13916 raft_consensus.cc:1261] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Refusing update from remote peer 60ba0342b9c045f68eed2932060444de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:08:59.422219 14202 raft_consensus.cc:1261] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 1 FOLLOWER]: Refusing update from remote peer 60ba0342b9c045f68eed2932060444de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260811 03:08:59.423139 14108 tablet.cc:2369] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:59.423381 14276 consensus_queue.cc:1035] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [LEADER]: Connected to new peer: Peer: permanent_uuid: "546615372ca340ad8326852c668e837e" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:08:59.423760 14272 consensus_queue.cc:1035] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fd7bd27874f4700a0fc11658f040935" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 03:08:59.436280 14250 tablet.cc:2369] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:08:59.437322 14278 mvcc.cc:205] Tried to move back new op lower bound from 7317167060661399552 to 7317167060517371904. Current Snapshot: MvccSnapshot[applied={T|T < 7317167060661399552}]
I20260811 03:08:59.450572 13757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:08:59.450958 13757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:08:59.455111 13757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Sending DeleteTablet for 3 replicas of tablet 8faf8cfc261d41c08cd04548ca151314
I20260811 03:08:59.456457 14039 tablet_service.cc:1514] Processing DeleteTablet for tablet 8faf8cfc261d41c08cd04548ca151314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:59 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:57348
I20260811 03:08:59.456434 13894 tablet_service.cc:1514] Processing DeleteTablet for tablet 8faf8cfc261d41c08cd04548ca151314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:59 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:39268
I20260811 03:08:59.457305 14180 tablet_service.cc:1514] Processing DeleteTablet for tablet 8faf8cfc261d41c08cd04548ca151314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:08:59 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:60188
I20260811 03:08:59.458577 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 13798
I20260811 03:08:59.467624 14290 tablet_replica.cc:330] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: stopping tablet replica
I20260811 03:08:59.469412 14290 raft_consensus.cc:2229] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:08:59.470183 14290 raft_consensus.cc:2258] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:59.471716 14289 tablet_replica.cc:330] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: stopping tablet replica
I20260811 03:08:59.472376 14289 raft_consensus.cc:2229] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:08:59.473109 14289 raft_consensus.cc:2258] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:59.474758 14288 tablet_replica.cc:330] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: stopping tablet replica
I20260811 03:08:59.475250 14288 raft_consensus.cc:2229] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:08:59.475943 14290 ts_tablet_manager.cc:1905] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:59.476034 14288 raft_consensus.cc:2258] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:08:59.478050 14288 ts_tablet_manager.cc:1905] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:59.478034 14289 ts_tablet_manager.cc:1905] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:08:59.482044 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.460183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.27.109.1:37181)
0811 03:08:59.460340 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.460345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.460360 (+ 15us) server_negotiation.cc:407] Connection header received
0811 03:08:59.460417 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.460420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.460487 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.460607 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:59.462643 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.463969 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.466108 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.466875 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.467047 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.471614 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:59.471636 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:59.471657 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:59.471722 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:59.476710 (+ 4988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.477949 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.477959 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.477962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.478372 (+ 410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.478935 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.478940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.478944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.479464 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:59.479822 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.481327 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.481605 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 03:08:59.483174 13958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:59.486814 14288 ts_tablet_manager.cc:1918] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:59.487142 14288 log.cc:1196] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/8faf8cfc261d41c08cd04548ca151314
I20260811 03:08:59.487897 14288 ts_tablet_manager.cc:1939] T 8faf8cfc261d41c08cd04548ca151314 P 60ba0342b9c045f68eed2932060444de: Deleting consensus metadata
I20260811 03:08:59.490458 13744 catalog_manager.cc:4841] TS 60ba0342b9c045f68eed2932060444de (127.27.109.2:43893): tablet 8faf8cfc261d41c08cd04548ca151314 (table test-table [id=77f16179803b46039b4e6e0fa271eec2]) successfully deleted
I20260811 03:08:59.496340 14290 ts_tablet_manager.cc:1918] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:08:59.496572 14290 log.cc:1196] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/8faf8cfc261d41c08cd04548ca151314
I20260811 03:08:59.497187 14290 ts_tablet_manager.cc:1939] T 8faf8cfc261d41c08cd04548ca151314 P 1fd7bd27874f4700a0fc11658f040935: Deleting consensus metadata
I20260811 03:08:59.499408 13745 catalog_manager.cc:4841] TS 1fd7bd27874f4700a0fc11658f040935 (127.27.109.3:42219): tablet 8faf8cfc261d41c08cd04548ca151314 (table test-table [id=77f16179803b46039b4e6e0fa271eec2]) successfully deleted
I20260811 03:08:59.599066 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 13798
W20260811 03:08:59.623287 13743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:37181 (error 108)
I20260811 03:08:59.623646 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 13967
I20260811 03:08:59.625674 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.625087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39284)
0811 03:08:59.625310 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.625467 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260811 03:08:59.626746 13743 catalog_manager.cc:4584] TS 546615372ca340ad8326852c668e837e (127.27.109.1:37181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8faf8cfc261d41c08cd04548ca151314: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
I20260811 03:08:59.641518 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.624575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.27.109.2:43893)
0811 03:08:59.624768 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.624774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.624799 (+ 25us) server_negotiation.cc:407] Connection header received
0811 03:08:59.625022 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.625027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.625121 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.625244 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:59.627065 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.628137 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.630189 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.630646 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.630930 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.634469 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:59.634484 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:59.634501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:59.634541 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:59.637945 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.638978 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.638986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.638988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.639337 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.639829 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.639833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.639835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.640291 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:59.640562 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.640885 (+ 323us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.641124 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 03:08:59.642393 14102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:59.680966 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.680343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39286)
0811 03:08:59.680623 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.680741 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260811 03:08:59.682204 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.681778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39292)
0811 03:08:59.681950 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.682042 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260811 03:08:59.742889 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.742258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39298)
0811 03:08:59.742496 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.742600 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260811 03:08:59.744314 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.743812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39310)
0811 03:08:59.744069 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.744163 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 03:08:59.771807 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 13967
I20260811 03:08:59.798355 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 14111
I20260811 03:08:59.816857 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.799499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.27.109.3:42219)
0811 03:08:59.799744 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 03:08:59.799751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:08:59.799774 (+ 23us) server_negotiation.cc:407] Connection header received
0811 03:08:59.799917 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:08:59.799923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:08:59.800023 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:08:59.800154 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:08:59.802016 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.803006 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.805178 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:08:59.805620 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:08:59.805742 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:08:59.809393 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:08:59.809412 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:08:59.809431 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:08:59.809495 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:08:59.813127 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.814314 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.814324 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.814326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.814535 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:08:59.815135 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:08:59.815139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:08:59.815141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:08:59.815616 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:08:59.815860 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0811 03:08:59.816226 (+ 366us) server_negotiation.cc:299] Negotiation successful
0811 03:08:59.816438 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 03:08:59.817627 14244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:08:59.825111 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.824552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39322)
0811 03:08:59.824786 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.824924 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260811 03:08:59.826241 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:08:59.825811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39326)
0811 03:08:59.826030 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 03:08:59.826120 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260811 03:08:59.913738 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 14111
I20260811 03:08:59.937362 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 13723
I20260811 03:08:59.938792 13791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:00.063180 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:00.062366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37181 (local address 127.0.0.1:39328)
0811 03:09:00.062568 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:00.062856 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260811 03:09:00.063480 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 13723
I20260811 03:09:20.092644 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41965
--webserver_interface=127.27.109.62
--webserver_port=44181
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:20.463145 14296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:20.463513 14296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:20.463677 14296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:20.471414 14296 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 03:09:20.471529 14296 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 03:09:20.471601 14296 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 03:09:20.471683 14296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:09:20.471784 14296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:20.471908 14296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:20.471961 14296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:09:20.472035 14296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:09:20.482563 14296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=44181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:20.484066 14296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:20.485672 14296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:20.498803 14304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:20.499018 14302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:20.499018 14301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:20.500244 14296 server_base.cc:1034] running on GCE node
I20260811 03:09:20.501185 14296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:20.502925 14296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:20.504184 14296 hybrid_clock.cc:648] HybridClock initialized: now 1786417760504096 us; error 53 us; skew 500 ppm
I20260811 03:09:20.504516 14296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:20 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417760, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:20.515167 14296 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:09:20.517131 14296 webserver.cc:460] Webserver started at http://127.27.109.62:44181/ using document root <none> and password file <none>
I20260811 03:09:20.517923 14296 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:20.518076 14296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:20.524415 14296 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:09:20.527832 14312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:20.529439 14296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:20.529680 14296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "3352b7df0d7f4338bb37b5ca7a28defe"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "4fdbfe19c22ded3d64e4a060efa53185"
server_key_iv: "3847231c39e8ccb8767d7ceef01cb62f"
server_key_version: "encryptionkey@0"
I20260811 03:09:20.530790 14296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:20.560683 14296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:20.565467 14296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:20.565801 14296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:20.587445 14296 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:41965
I20260811 03:09:20.587497 14367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:41965 every 8 connection(s)
I20260811 03:09:20.589109 14296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:09:20.596670 14368 sys_catalog.cc:263] Verifying existing consensus state
I20260811 03:09:20.597731 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 14296
I20260811 03:09:20.598843 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:37181
--local_ip_for_outbound_sockets=127.27.109.1
--tserver_master_addrs=127.27.109.62:41965
--webserver_port=36937
--webserver_interface=127.27.109.1
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:09:20.602027 14368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Bootstrap starting.
I20260811 03:09:20.626670 14368 log.cc:824] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:20.644706 14368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 03:09:20.645346 14368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Bootstrap complete.
I20260811 03:09:20.654914 14368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:09:20.655984 14368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3352b7df0d7f4338bb37b5ca7a28defe, State: Initialized, Role: FOLLOWER
I20260811 03:09:20.656663 14368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:09:20.656926 14368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:09:20.657079 14368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:09:20.657279 14368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 2 FOLLOWER]: Advancing to term 3
I20260811 03:09:20.661656 14368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:09:20.662242 14368 leader_election.cc:302] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3352b7df0d7f4338bb37b5ca7a28defe; no voters:
I20260811 03:09:20.663059 14368 leader_election.cc:288] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 03:09:20.663218 14372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 3 FOLLOWER]: Leader election won for term 3
I20260811 03:09:20.664538 14372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [term 3 LEADER]: Becoming Leader. State: Replica: 3352b7df0d7f4338bb37b5ca7a28defe, State: Running, Role: LEADER
I20260811 03:09:20.665335 14372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } }
I20260811 03:09:20.665534 14368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:09:20.668078 14374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3352b7df0d7f4338bb37b5ca7a28defe. Latest consensus state: current_term: 3 leader_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } } }
I20260811 03:09:20.668557 14374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:20.669217 14373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3352b7df0d7f4338bb37b5ca7a28defe" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 41965 } } }
I20260811 03:09:20.669620 14373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:20.676306 14379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:09:20.685408 14379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:09:20.687283 14379 catalog_manager.cc:1259] Loaded cluster ID: cbdf9e92667f4fd9979214722d56aef4
I20260811 03:09:20.687497 14379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:09:20.692461 14379 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:09:20.694618 14379 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Loaded TSK: 0
I20260811 03:09:20.694962 14379 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3352b7df0d7f4338bb37b5ca7a28defe: Loaded TSK: 1
F20260811 03:09:20.696343 14379 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f55b841c9ad google::LogMessage::Fail() at ??:0
@ 0x7f55b841f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f55b841c4de google::LogMessage::Flush() at ??:0
@ 0x7f55b841da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f55bc1807d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f55bc16f73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f55cde4b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f55cde4a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f55cded62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f55cded60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f55cdf205e9 std::function<>::operator()() at ??:0
@ 0x7f55cde4ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f55cde4cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f55cded4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f55cded4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f55bb068345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f55bb06b91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f55bb06b799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f55bb046eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f55be3206db start_thread at ??:0
@ 0x7f55b72b661f clone at ??:0
W20260811 03:09:21.022217 14370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:21.022629 14370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:21.022864 14370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:21.030561 14370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:21.030778 14370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:21.030963 14370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:09:21.041450 14370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:37181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=36937
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:21.043110 14370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:21.044598 14370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:21.058624 14421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:21.058838 14422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:21.059229 14424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:21.060192 14370 server_base.cc:1034] running on GCE node
I20260811 03:09:21.060873 14370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:21.061908 14370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:21.063150 14370 hybrid_clock.cc:648] HybridClock initialized: now 1786417761063097 us; error 47 us; skew 500 ppm
I20260811 03:09:21.063501 14370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:21 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417761, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:21.066886 14370 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:09:21.068478 14370 webserver.cc:460] Webserver started at http://127.27.109.1:36937/ using document root <none> and password file <none>
I20260811 03:09:21.069188 14370 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:21.069295 14370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:21.075156 14370 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 03:09:21.078475 14431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:21.080201 14370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:21.080451 14370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "546615372ca340ad8326852c668e837e"
format_stamp: "Formatted at 2026-08-11 03:08:34 on dist-test-slave-8tdl"
server_key: "9e0540de41198c17bc45fc8989c32a3a"
server_key_iv: "367a8104a18f7fae04b296c93d9810fd"
server_key_version: "encryptionkey@0"
I20260811 03:09:21.081574 14370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:21.099601 14370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:21.104668 14370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:21.104975 14370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:21.106166 14370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:21.108281 14438 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260811 03:09:21.114113 14438 ts_tablet_manager.cc:1852] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260811 03:09:21.115639 14438 ts_tablet_manager.cc:1905] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:21.123626 14438 ts_tablet_manager.cc:1918] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:21.123823 14438 log.cc:1196] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/8faf8cfc261d41c08cd04548ca151314
I20260811 03:09:21.124284 14438 ts_tablet_manager.cc:1939] T 8faf8cfc261d41c08cd04548ca151314 P 546615372ca340ad8326852c668e837e: Deleting consensus metadata
I20260811 03:09:21.124724 14370 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260811 03:09:21.124912 14370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260811 03:09:21.125080 14370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:21.125178 14370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:21.162884 14370 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37181
I20260811 03:09:21.162976 14552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37181 every 8 connection(s)
I20260811 03:09:21.164292 14370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:09:21.166059 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 14370
I20260811 03:09:21.166792 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:43893
--local_ip_for_outbound_sockets=127.27.109.2
--tserver_master_addrs=127.27.109.62:41965
--webserver_port=41917
--webserver_interface=127.27.109.2
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:09:21.172077 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:21.167730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:37155)
0811 03:09:21.170509 (+ 2779us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:21.170870 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2419,"thread_start_us":2185,"threads_started":1}
W20260811 03:09:21.173132 14553 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:41965 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:41965: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
I20260811 03:09:21.174345 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:21.173810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:38409)
0811 03:09:21.174074 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:21.174179 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 03:09:21.175865 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:21.175233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:54861)
0811 03:09:21.175451 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:21.175608 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 03:09:21.176230 14553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 03:09:21.571168 14556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:21.571481 14556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:21.571708 14556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:21.579761 14556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:21.579903 14556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:21.580175 14556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:09:21.590993 14556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:43893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=41917
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:21.592604 14556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:21.594388 14556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:21.609066 14561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:21.609089 14562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:21.609400 14564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:21.610059 14556 server_base.cc:1034] running on GCE node
I20260811 03:09:21.610760 14556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:21.611898 14556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:21.613125 14556 hybrid_clock.cc:648] HybridClock initialized: now 1786417761613080 us; error 32 us; skew 500 ppm
I20260811 03:09:21.613481 14556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:21 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417761, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:21.616937 14556 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:09:21.618471 14556 webserver.cc:460] Webserver started at http://127.27.109.2:41917/ using document root <none> and password file <none>
I20260811 03:09:21.619254 14556 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:21.619407 14556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:21.624926 14556 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 03:09:21.627935 14572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:21.629493 14556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 03:09:21.629707 14556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "60ba0342b9c045f68eed2932060444de"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "3b2632bb99eafcff8f03fcda983257b2"
server_key_iv: "ef9c57ed3d88be5396a33e48d5b22385"
server_key_version: "encryptionkey@0"
I20260811 03:09:21.630874 14556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:21.651700 14556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:21.656572 14556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:21.656991 14556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:21.658165 14556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:21.659827 14556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:21.659988 14556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:21.660146 14556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:21.660249 14556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:21.698009 14556 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:43893
I20260811 03:09:21.698105 14691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:43893 every 8 connection(s)
I20260811 03:09:21.699736 14556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:09:21.705112 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 14556
I20260811 03:09:21.705132 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:21.703070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:47185)
0811 03:09:21.703844 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:21.704184 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":302,"threads_started":1}
W20260811 03:09:21.705919 14692 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:41965 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:41965: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
I20260811 03:09:21.705935 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:42219
--local_ip_for_outbound_sockets=127.27.109.3
--tserver_master_addrs=127.27.109.62:41965
--webserver_port=40835
--webserver_interface=127.27.109.3
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:09:21.707062 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:21.706575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:48483)
0811 03:09:21.706799 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:21.706877 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 03:09:21.708187 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:21.707749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:43083)
0811 03:09:21.707926 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:21.708049 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260811 03:09:21.708484 14692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 03:09:22.064646 14695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:22.064980 14695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:22.065151 14695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:22.072837 14695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:22.073063 14695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:22.073285 14695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:09:22.083595 14695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:42219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=40835
--tserver_master_addrs=127.27.109.62:41965
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:22.085002 14695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:22.086557 14695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:22.103097 14701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:22.103183 14702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:22.103171 14704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:22.104619 14695 server_base.cc:1034] running on GCE node
I20260811 03:09:22.105378 14695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:22.106519 14695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:22.107738 14695 hybrid_clock.cc:648] HybridClock initialized: now 1786417762107629 us; error 103 us; skew 500 ppm
I20260811 03:09:22.108114 14695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:22 dist-test-slave-8tdl krb5kdc[13115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417762, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:22.111701 14695 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:09:22.113354 14695 webserver.cc:460] Webserver started at http://127.27.109.3:40835/ using document root <none> and password file <none>
I20260811 03:09:22.114104 14695 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:22.114246 14695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:22.120308 14695 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260811 03:09:22.123675 14711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:22.125368 14695 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260811 03:09:22.125607 14695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "1fd7bd27874f4700a0fc11658f040935"
format_stamp: "Formatted at 2026-08-11 03:08:35 on dist-test-slave-8tdl"
server_key: "411f7a672f57e8f870222fe5512daa96"
server_key_iv: "7af77c0c4995df10feaa3497057b57bc"
server_key_version: "encryptionkey@0"
I20260811 03:09:22.126785 14695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:22.147675 14695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:22.152827 14695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:22.153163 14695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:22.154336 14695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:22.156028 14695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:22.156167 14695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:22.156344 14695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:22.156438 14695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:22.178038 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:22.176994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:60547)
0811 03:09:22.177552 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:22.177655 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":272,"threads_started":1}
I20260811 03:09:22.195297 14695 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:42219
I20260811 03:09:22.195350 14833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:42219 every 8 connection(s)
I20260811 03:09:22.196928 14695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:09:22.202317 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:22.200191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:38327)
0811 03:09:22.200972 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:22.201397 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":313,"threads_started":1}
I20260811 03:09:22.202669 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 14695
W20260811 03:09:22.203205 14834 heartbeater.cc:644] Failed to heartbeat to 127.27.109.62:41965 (0 consecutive failures): Network error: Failed to ping master at 127.27.109.62:41965: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
I20260811 03:09:22.204138 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:22.203741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:52453)
0811 03:09:22.203902 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:22.203987 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260811 03:09:22.205425 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:22.205050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:41157)
0811 03:09:22.205198 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:22.205274 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260811 03:09:22.205724 14834 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 03:09:22.709931 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:22.709085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:49539)
0811 03:09:22.709568 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:22.709690 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":224,"threads_started":1}
I20260811 03:09:23.180012 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:23.179241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:39579)
0811 03:09:23.179717 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:23.179806 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":247,"threads_started":1}
I20260811 03:09:23.207373 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:23.206426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:46749)
0811 03:09:23.206979 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:23.207133 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
I20260811 03:09:23.711947 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:23.711062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:60555)
0811 03:09:23.711675 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:23.711760 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":310,"threads_started":1}
I20260811 03:09:24.181695 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:24.180925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:45611)
0811 03:09:24.181413 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:24.181510 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":238,"threads_started":1}
I20260811 03:09:24.209442 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:24.208456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:58567)
0811 03:09:24.209117 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:24.209230 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":335,"threads_started":1}
I20260811 03:09:24.713876 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:24.713018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:36353)
0811 03:09:24.713573 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:24.713679 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":286,"threads_started":1}
I20260811 03:09:25.183812 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:25.182945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:47367)
0811 03:09:25.183529 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:25.183629 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":283,"threads_started":1}
I20260811 03:09:25.211591 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:25.210604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:51981)
0811 03:09:25.211300 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:25.211395 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":421,"threads_started":1}
I20260811 03:09:25.715816 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:25.715030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:36071)
0811 03:09:25.715526 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:25.715623 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":241,"threads_started":1}
I20260811 03:09:26.185920 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:26.185018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:47639)
0811 03:09:26.185642 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:26.185733 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":384,"threads_started":1}
I20260811 03:09:26.213337 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:26.212547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:60147)
0811 03:09:26.213063 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:26.213157 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":252,"threads_started":1}
I20260811 03:09:26.717722 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:26.716917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:53325)
0811 03:09:26.717433 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:26.717545 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":260,"threads_started":1}
I20260811 03:09:27.188069 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:27.187122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:48721)
0811 03:09:27.187766 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:27.187869 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":361,"threads_started":1}
I20260811 03:09:27.215423 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:27.214533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:58915)
0811 03:09:27.215147 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:27.215239 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":339,"threads_started":1}
I20260811 03:09:27.719611 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:27.718769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:51737)
0811 03:09:27.719318 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:27.719430 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":264,"threads_started":1}
I20260811 03:09:28.190244 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:28.189174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:33091)
0811 03:09:28.189846 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:28.190007 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":360,"threads_started":1}
I20260811 03:09:28.217386 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:28.216571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:36159)
0811 03:09:28.217099 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:28.217205 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":235,"threads_started":1}
I20260811 03:09:28.721453 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:28.720606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:51559)
0811 03:09:28.721141 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:28.721234 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":223,"threads_started":1}
I20260811 03:09:29.191948 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:29.191220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:44963)
0811 03:09:29.191696 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:29.191775 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
I20260811 03:09:29.219316 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:29.218411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:45361)
0811 03:09:29.219000 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:29.219095 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260811 03:09:29.723730 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:29.722647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:33793)
0811 03:09:29.723305 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:29.723456 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":258,"threads_started":1}
I20260811 03:09:30.193647 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:30.192850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:51647)
0811 03:09:30.193366 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:30.193467 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":258,"threads_started":1}
I20260811 03:09:30.221108 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:30.220361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:33673)
0811 03:09:30.220840 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:30.220929 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260811 03:09:30.725824 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:30.724858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:34693)
0811 03:09:30.725492 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:30.725585 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":353,"threads_started":1}
I20260811 03:09:31.195477 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:31.194741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:47005)
0811 03:09:31.195213 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:31.195303 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
I20260811 03:09:31.223073 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:31.222192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:40417)
0811 03:09:31.222763 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:31.222878 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":247,"threads_started":1}
I20260811 03:09:31.727967 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:31.726972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:54489)
0811 03:09:31.727600 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:31.727720 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":325,"threads_started":1}
I20260811 03:09:32.197817 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:32.196683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:45799)
0811 03:09:32.197449 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:32.197565 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":476,"threads_started":1}
I20260811 03:09:32.224915 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:32.224113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:56851)
0811 03:09:32.224646 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:32.224738 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260811 03:09:32.729871 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:32.729062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:55703)
0811 03:09:32.729548 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:32.729678 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":243,"threads_started":1}
I20260811 03:09:33.200119 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:33.199165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:36565)
0811 03:09:33.199795 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:33.199922 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":328,"threads_started":1}
I20260811 03:09:33.227027 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:33.225944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:56197)
0811 03:09:33.226632 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:33.226803 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":264,"threads_started":1}
I20260811 03:09:33.732079 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:33.731129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:42279)
0811 03:09:33.731712 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:33.731847 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":269,"threads_started":1}
I20260811 03:09:34.202324 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:34.201406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:60167)
0811 03:09:34.201929 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:34.202130 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":239,"threads_started":1}
I20260811 03:09:34.228950 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:34.228112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:51399)
0811 03:09:34.228634 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:34.228725 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":285,"threads_started":1}
I20260811 03:09:34.733934 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:34.733047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:50091)
0811 03:09:34.733614 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:34.733726 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":245,"threads_started":1}
I20260811 03:09:35.204245 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:35.203354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:41513)
0811 03:09:35.203900 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:35.204044 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":265,"threads_started":1}
I20260811 03:09:35.230741 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:35.229921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:49805)
0811 03:09:35.230413 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:35.230522 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":248,"threads_started":1}
I20260811 03:09:35.743635 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:35.735169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:32957)
0811 03:09:35.743220 (+ 8051us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:35.743380 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7877,"thread_start_us":7706,"threads_started":1}
I20260811 03:09:36.206291 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:36.205452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:58965)
0811 03:09:36.205990 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:36.206092 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":249,"threads_started":1}
I20260811 03:09:36.232528 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:36.231754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:43831)
0811 03:09:36.232238 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:36.232326 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":253,"threads_started":1}
I20260811 03:09:36.745908 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:36.745062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.2:34173)
0811 03:09:36.745587 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:36.745692 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260811 03:09:37.208630 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:37.207668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.1:50589)
0811 03:09:37.208341 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:37.208446 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":392,"threads_started":1}
I20260811 03:09:37.209365 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 14370
I20260811 03:09:37.230541 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:37.210855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.27.109.1:37181)
0811 03:09:37.212120 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:37.212134 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:37.212172 (+ 38us) server_negotiation.cc:407] Connection header received
0811 03:09:37.213994 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:37.214005 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:37.214479 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:37.214880 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:37.216450 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:37.217807 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:37.219355 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:37.220100 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:37.220259 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:37.223480 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:37.223488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:37.223506 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:37.223555 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:37.227006 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:37.227872 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:37.227879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:37.227887 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:37.228101 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:37.228450 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:37.228454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:37.228455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:37.229014 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:37.229323 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:37.229766 (+ 443us) server_negotiation.cc:299] Negotiation successful
0811 03:09:37.230227 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":918,"threads_started":1}
I20260811 03:09:37.232090 14547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:37.239434 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:37.233890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:41965 (local address 127.27.109.3:56559)
0811 03:09:37.239121 (+ 5231us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:37.239232 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.62:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5070,"thread_start_us":4958,"threads_started":1}
I20260811 03:09:37.324427 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 14370
I20260811 03:09:37.348119 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 14556
I20260811 03:09:37.368317 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:37.349190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.27.109.2:43893)
0811 03:09:37.349713 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:37.349728 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:37.349768 (+ 40us) server_negotiation.cc:407] Connection header received
0811 03:09:37.351524 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:37.351535 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:37.351980 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:37.352305 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:37.353903 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:37.355540 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:37.356954 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:37.357686 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:37.357838 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:37.361246 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:37.361255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:37.361273 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:37.361319 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:37.364702 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:37.365592 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:37.365599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:37.365606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:37.365790 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:37.366219 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:37.366222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:37.366224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:37.366859 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:37.367140 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:37.367621 (+ 481us) server_negotiation.cc:299] Negotiation successful
0811 03:09:37.368018 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":225,"threads_started":1}
I20260811 03:09:37.369874 14686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:37.456219 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 14556
I20260811 03:09:37.479569 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 14695
I20260811 03:09:37.499307 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:37.480883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.27.109.3:42219)
0811 03:09:37.481509 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:37.481525 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:37.481573 (+ 48us) server_negotiation.cc:407] Connection header received
0811 03:09:37.483411 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:37.483423 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:37.483883 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:37.484263 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:37.485814 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:37.487168 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:37.488462 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:37.489161 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:37.489360 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:37.492372 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:37.492380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:37.492397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:37.492441 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:37.495735 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:37.496466 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:37.496479 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:37.496487 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:37.496674 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:37.497085 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:37.497095 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:37.497097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:37.497691 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:37.497996 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:37.498482 (+ 486us) server_negotiation.cc:299] Negotiation successful
0811 03:09:37.498982 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":292,"threads_started":1}
I20260811 03:09:37.500986 14829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:37.593345 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 14695
2026-08-11T03:09:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65495 ms)
[----------] 30 tests from SecurityITest (323430 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:09:37 dist-test-slave-8tdl krb5kdc[14897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:09:37 dist-test-slave-8tdl krb5kdc[14897](info): set up 2 sockets
Aug 11 03:09:37 dist-test-slave-8tdl krb5kdc[14897](info): commencing operation
krb5kdc: starting...
W20260811 03:09:38.797950 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:09:39 dist-test-slave-8tdl krb5kdc[14897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:09:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:09:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:09:39.146749 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32935
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:32935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:39.520874 14910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:39.521294 14910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:39.521459 14910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:39.529744 14910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:09:39.529907 14910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:39.530004 14910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:39.530090 14910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:09:39.530160 14910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:09:39.541225 14910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:32935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:32935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:39.542897 14910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:39.544361 14910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:39.558128 14915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:39.557986 14918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:39.558034 14916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:39.559037 14910 server_base.cc:1034] running on GCE node
I20260811 03:09:39.560170 14910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:39.561877 14910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:39.563127 14910 hybrid_clock.cc:648] HybridClock initialized: now 1786417779563067 us; error 55 us; skew 500 ppm
I20260811 03:09:39.563546 14910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:39 dist-test-slave-8tdl krb5kdc[14897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417779, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:39.572585 14910 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:09:39.574605 14910 webserver.cc:460] Webserver started at http://127.27.109.62:38759/ using document root <none> and password file <none>
I20260811 03:09:39.575464 14910 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:39.575608 14910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:39.575976 14910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:39.579316 14910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "61596738757c4d709c13aa41cef54769"
format_stamp: "Formatted at 2026-08-11 03:09:39 on dist-test-slave-8tdl"
server_key: "b5cb215efebd3d37e72e451a7a7d0749"
server_key_iv: "20d3c4b9c2f38c9be2c86cbc75d593b3"
server_key_version: "encryptionkey@0"
I20260811 03:09:39.580202 14910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "61596738757c4d709c13aa41cef54769"
format_stamp: "Formatted at 2026-08-11 03:09:39 on dist-test-slave-8tdl"
server_key: "b5cb215efebd3d37e72e451a7a7d0749"
server_key_iv: "20d3c4b9c2f38c9be2c86cbc75d593b3"
server_key_version: "encryptionkey@0"
I20260811 03:09:39.586023 14910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:09:39.590560 14925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:39.592264 14910 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260811 03:09:39.592531 14910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "61596738757c4d709c13aa41cef54769"
format_stamp: "Formatted at 2026-08-11 03:09:39 on dist-test-slave-8tdl"
server_key: "b5cb215efebd3d37e72e451a7a7d0749"
server_key_iv: "20d3c4b9c2f38c9be2c86cbc75d593b3"
server_key_version: "encryptionkey@0"
I20260811 03:09:39.592792 14910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:39.630116 14910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:39.635223 14910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:39.635566 14910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:39.656889 14910 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:32935
I20260811 03:09:39.656991 14981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:32935 every 8 connection(s)
I20260811 03:09:39.658432 14910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:09:39.662328 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 14910
I20260811 03:09:39.662609 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:09:39.663020 14982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:39.663578 28084 external_mini_cluster.cc:1428] Setting key 9fe10b74d497171dcd046f3050572d63
I20260811 03:09:39.679282 14982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769: Bootstrap starting.
Aug 11 03:09:39 dist-test-slave-8tdl krb5kdc[14897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:39.684041 14982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:39.685767 14982 log.cc:824] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:39.690531 14982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769: No bootstrap required, opened a new log
I20260811 03:09:39.691422 14986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:39.665876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.27.109.62:32935)
0811 03:09:39.666812 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:39.666824 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:39.666878 (+ 54us) server_negotiation.cc:407] Connection header received
0811 03:09:39.668673 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:39.668698 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:39.669124 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:39.669489 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:39.671531 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:39.672899 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:39.674819 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:39.675622 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:39.675766 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:39.681338 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:39.681355 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:39.681379 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:39.681460 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:39.685388 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:39.686320 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:39.686330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:39.686339 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:39.686623 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:39.687218 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:39.687223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:39.687226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:39.687827 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:39.688136 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:39.689451 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0811 03:09:39.690005 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":464,"threads_started":1}
I20260811 03:09:39.700738 14982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61596738757c4d709c13aa41cef54769" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32935 } }
I20260811 03:09:39.701205 14982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:39.701295 14982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61596738757c4d709c13aa41cef54769, State: Initialized, Role: FOLLOWER
I20260811 03:09:39.701807 14982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61596738757c4d709c13aa41cef54769" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32935 } }
I20260811 03:09:39.702051 14982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:09:39.702214 14982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:09:39.702404 14982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:39.704960 14982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61596738757c4d709c13aa41cef54769" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32935 } }
I20260811 03:09:39.705536 14982 leader_election.cc:302] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61596738757c4d709c13aa41cef54769; no voters:
I20260811 03:09:39.706192 14982 leader_election.cc:288] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:09:39.706364 14988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:39.707144 14988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [term 1 LEADER]: Becoming Leader. State: Replica: 61596738757c4d709c13aa41cef54769, State: Running, Role: LEADER
I20260811 03:09:39.707729 14988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61596738757c4d709c13aa41cef54769" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32935 } }
I20260811 03:09:39.708458 14982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:09:39.711455 14990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61596738757c4d709c13aa41cef54769. Latest consensus state: current_term: 1 leader_uuid: "61596738757c4d709c13aa41cef54769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61596738757c4d709c13aa41cef54769" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32935 } } }
I20260811 03:09:39.711504 14989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61596738757c4d709c13aa41cef54769" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61596738757c4d709c13aa41cef54769" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 32935 } } }
I20260811 03:09:39.711925 14990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:39.711952 14989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769 [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:39.712958 14996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:09:39.719595 14996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:09:39.733106 14996 catalog_manager.cc:1347] Generated new cluster ID: 26b88b466ed34188960a27575a9c3d82
I20260811 03:09:39.733263 14996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:09:39.752096 14996 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:09:39.753558 14996 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:09:39.760936 14996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769: Generated new TSK 0
I20260811 03:09:39.761919 14996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:09:39.858621 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32935
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:40.256126 15010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:40.256471 15010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:40.256575 15010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:40.264676 15010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:40.264806 15010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:40.264927 15010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:09:40.275388 15010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:40.276826 15010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:40.278224 15010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:40.294082 15018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:40.294257 15015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:40.294502 15016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:40.295619 15010 server_base.cc:1034] running on GCE node
I20260811 03:09:40.296516 15010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:40.297775 15010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:40.299022 15010 hybrid_clock.cc:648] HybridClock initialized: now 1786417780298922 us; error 86 us; skew 500 ppm
I20260811 03:09:40.299386 15010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:40 dist-test-slave-8tdl krb5kdc[14897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:40.303176 15010 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:09:40.304943 15010 webserver.cc:460] Webserver started at http://127.27.109.1:36603/ using document root <none> and password file <none>
I20260811 03:09:40.305701 15010 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:40.305840 15010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:40.306226 15010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:40.309644 15010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "8a0d8cadf23844058e0f1ff941b46767"
format_stamp: "Formatted at 2026-08-11 03:09:40 on dist-test-slave-8tdl"
server_key: "c3d5d40de17443f676bc0631a9719785"
server_key_iv: "9bcca07819ebd1e2edabee932c56c6a3"
server_key_version: "encryptionkey@0"
I20260811 03:09:40.310570 15010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "8a0d8cadf23844058e0f1ff941b46767"
format_stamp: "Formatted at 2026-08-11 03:09:40 on dist-test-slave-8tdl"
server_key: "c3d5d40de17443f676bc0631a9719785"
server_key_iv: "9bcca07819ebd1e2edabee932c56c6a3"
server_key_version: "encryptionkey@0"
I20260811 03:09:40.316524 15010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260811 03:09:40.320423 15025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:40.322103 15010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260811 03:09:40.322310 15010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "8a0d8cadf23844058e0f1ff941b46767"
format_stamp: "Formatted at 2026-08-11 03:09:40 on dist-test-slave-8tdl"
server_key: "c3d5d40de17443f676bc0631a9719785"
server_key_iv: "9bcca07819ebd1e2edabee932c56c6a3"
server_key_version: "encryptionkey@0"
I20260811 03:09:40.322570 15010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:40.358779 15010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:40.364177 15010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:40.364476 15010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:40.365522 15010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:40.367139 15010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:40.367220 15010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:40.367321 15010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:40.367374 15010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:40.404448 15010 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37551
I20260811 03:09:40.404557 15145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37551 every 8 connection(s)
I20260811 03:09:40.405908 15010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:09:40.407356 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 15010
I20260811 03:09:40.407629 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:09:40.408524 28084 external_mini_cluster.cc:1428] Setting key e9fffe27cb5e69dc5c962c1b835bbdaf
Aug 11 03:09:40 dist-test-slave-8tdl krb5kdc[14897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:40.431105 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:40.409689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:46577 (local address 127.27.109.62:32935)
0811 03:09:40.410754 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:40.410761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:40.411699 (+ 938us) server_negotiation.cc:407] Connection header received
0811 03:09:40.413688 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:40.413696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:40.413791 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:40.413915 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:40.415753 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:40.416911 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:40.418559 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:40.419315 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:40.419480 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:40.423875 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:40.423885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:40.423893 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:40.423948 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:40.427610 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:40.428532 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:40.428539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:40.428542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:40.428692 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:40.429174 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:40.429179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:40.429183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:40.429477 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:40.429643 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:40.430405 (+ 762us) server_negotiation.cc:299] Negotiation successful
0811 03:09:40.430661 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":767,"threads_started":1}
I20260811 03:09:40.432423 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:40.410569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32935 (local address 127.27.109.1:46577)
0811 03:09:40.411505 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:40.411556 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:40.413383 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:40.414230 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:40.414239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:40.414925 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:40.415577 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:40.415597 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:40.417088 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:40.417092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:40.418402 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:40.418415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:40.419611 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:40.419618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:40.419976 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:40.420787 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:40.420817 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:40.423563 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:40.427799 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:40.427817 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:40.427834 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:40.428371 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:40.428817 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:40.428822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:40.428826 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:40.429044 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:40.429617 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:40.429625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:40.430007 (+ 382us) client_negotiation.cc:769] Sending connection context
0811 03:09:40.430296 (+ 289us) client_negotiation.cc:240] Negotiation successful
0811 03:09:40.430931 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":389,"threads_started":1}
I20260811 03:09:40.434278 15146 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32935
I20260811 03:09:40.434614 15146 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:40.435449 15146 heartbeater.cc:511] Master 127.27.109.62:32935 requested a full tablet report, sending...
I20260811 03:09:40.437950 14944 ts_manager.cc:194] Registered new tserver with Master: 8a0d8cadf23844058e0f1ff941b46767 (127.27.109.1:37551)
I20260811 03:09:40.440305 14944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:46577
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:09:40.487349 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32935
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:40.868425 15152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:40.868757 15152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:40.868865 15152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:40.877386 15152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:40.877537 15152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:40.877671 15152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:09:40.889009 15152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:40.890509 15152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:40.892391 15152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:40.908766 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:40.908723 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:40.908682 15160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:40.909696 15152 server_base.cc:1034] running on GCE node
I20260811 03:09:40.910429 15152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:40.911594 15152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:40.912889 15152 hybrid_clock.cc:648] HybridClock initialized: now 1786417780912833 us; error 39 us; skew 500 ppm
I20260811 03:09:40.913300 15152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:40 dist-test-slave-8tdl krb5kdc[14897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:40.917207 15152 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:09:40.919003 15152 webserver.cc:460] Webserver started at http://127.27.109.2:40211/ using document root <none> and password file <none>
I20260811 03:09:40.919749 15152 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:40.919872 15152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:40.920161 15152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:40.923386 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "401bda4c4e92419a9fd53db21051e2ef"
format_stamp: "Formatted at 2026-08-11 03:09:40 on dist-test-slave-8tdl"
server_key: "c476ba4956d45b1eb92c6579e1979ef7"
server_key_iv: "73cbfd0410bc3fd33a27d874b390f3eb"
server_key_version: "encryptionkey@0"
I20260811 03:09:40.924182 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "401bda4c4e92419a9fd53db21051e2ef"
format_stamp: "Formatted at 2026-08-11 03:09:40 on dist-test-slave-8tdl"
server_key: "c476ba4956d45b1eb92c6579e1979ef7"
server_key_iv: "73cbfd0410bc3fd33a27d874b390f3eb"
server_key_version: "encryptionkey@0"
I20260811 03:09:40.930214 15152 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 03:09:40.934666 15168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:40.936450 15152 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:09:40.936633 15152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "401bda4c4e92419a9fd53db21051e2ef"
format_stamp: "Formatted at 2026-08-11 03:09:40 on dist-test-slave-8tdl"
server_key: "c476ba4956d45b1eb92c6579e1979ef7"
server_key_iv: "73cbfd0410bc3fd33a27d874b390f3eb"
server_key_version: "encryptionkey@0"
I20260811 03:09:40.936837 15152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:40.961874 15152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:40.967347 15152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:40.967734 15152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:40.968905 15152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:40.970492 15152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:40.970578 15152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:40.970757 15152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:40.970820 15152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260811 03:09:41.008062 15152 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:37303
I20260811 03:09:41.008234 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:37303 every 8 connection(s)
I20260811 03:09:41.009677 15152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:09:41.013720 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 15152
I20260811 03:09:41.014014 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:09:41.014916 28084 external_mini_cluster.cc:1428] Setting key ee5c90637cfe713493064f53cbbdb4dd
Aug 11 03:09:41 dist-test-slave-8tdl krb5kdc[14897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:41.037889 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.013024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:43571 (local address 127.27.109.62:32935)
0811 03:09:41.013671 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.013679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.015625 (+ 1946us) server_negotiation.cc:407] Connection header received
0811 03:09:41.018362 (+ 2737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.018368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.018514 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.018714 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:41.020640 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.021981 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.023943 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.024779 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.024970 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.029535 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:41.029546 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:41.029554 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:41.029612 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:41.033783 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:41.035058 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:41.035063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:41.035065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:41.035218 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:41.035866 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:41.035870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:41.035873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:41.036226 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:41.036368 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.037359 (+ 991us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.037599 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":271,"threads_started":1}
I20260811 03:09:41.039587 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.013699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32935 (local address 127.27.109.2:43571)
0811 03:09:41.015375 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:41.015446 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:41.017912 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:41.019148 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:41.019157 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:41.019753 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:41.020417 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.020438 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.022207 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.022212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.023753 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.023768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.025079 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.025085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.025319 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.026140 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:41.026172 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:41.029167 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:41.033975 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:41.033989 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:41.034014 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:41.034831 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:41.035355 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:41.035363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:41.035369 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:41.035656 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:41.036382 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:41.036393 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:41.036903 (+ 510us) client_negotiation.cc:769] Sending connection context
0811 03:09:41.037230 (+ 327us) client_negotiation.cc:240] Negotiation successful
0811 03:09:41.037921 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1344,"thread_start_us":981,"threads_started":1}
I20260811 03:09:41.041409 15290 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32935
I20260811 03:09:41.041822 15290 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:41.042953 15290 heartbeater.cc:511] Master 127.27.109.62:32935 requested a full tablet report, sending...
I20260811 03:09:41.044816 14944 ts_manager.cc:194] Registered new tserver with Master: 401bda4c4e92419a9fd53db21051e2ef (127.27.109.2:37303)
I20260811 03:09:41.046206 14944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:43571
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:09:41.099620 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32935
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:09:41.443955 15146 heartbeater.cc:503] Master 127.27.109.62:32935 was elected leader, sending a full tablet report...
W20260811 03:09:41.473673 15296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:41.474005 15296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:41.474210 15296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:41.481835 15296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:41.481982 15296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:41.482231 15296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:09:41.492527 15296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:32935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:41.493917 15296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:41.495450 15296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:41.509959 15302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:41.510862 15303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:41.510887 15305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:41.511781 15296 server_base.cc:1034] running on GCE node
I20260811 03:09:41.512588 15296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:41.513765 15296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:41.515039 15296 hybrid_clock.cc:648] HybridClock initialized: now 1786417781514995 us; error 52 us; skew 500 ppm
I20260811 03:09:41.515491 15296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:41 dist-test-slave-8tdl krb5kdc[14897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417781, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:41.518976 15296 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:09:41.520756 15296 webserver.cc:460] Webserver started at http://127.27.109.3:34789/ using document root <none> and password file <none>
I20260811 03:09:41.521553 15296 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:41.521716 15296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:41.522086 15296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:41.525454 15296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "01fa4d2a02a94fcf8895ebc5e78f6482"
format_stamp: "Formatted at 2026-08-11 03:09:41 on dist-test-slave-8tdl"
server_key: "1fe15da9c5c35a352671baff3985d26b"
server_key_iv: "5603b9b8fa2c86a26a164aaadb6f4a1b"
server_key_version: "encryptionkey@0"
I20260811 03:09:41.526325 15296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "01fa4d2a02a94fcf8895ebc5e78f6482"
format_stamp: "Formatted at 2026-08-11 03:09:41 on dist-test-slave-8tdl"
server_key: "1fe15da9c5c35a352671baff3985d26b"
server_key_iv: "5603b9b8fa2c86a26a164aaadb6f4a1b"
server_key_version: "encryptionkey@0"
I20260811 03:09:41.532279 15296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 03:09:41.536448 15312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:41.539004 15296 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260811 03:09:41.539227 15296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "01fa4d2a02a94fcf8895ebc5e78f6482"
format_stamp: "Formatted at 2026-08-11 03:09:41 on dist-test-slave-8tdl"
server_key: "1fe15da9c5c35a352671baff3985d26b"
server_key_iv: "5603b9b8fa2c86a26a164aaadb6f4a1b"
server_key_version: "encryptionkey@0"
I20260811 03:09:41.539486 15296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:41.556524 15296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:41.561625 15296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:41.561897 15296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:41.563017 15296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:41.564512 15296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:41.564594 15296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:41.564684 15296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:41.564733 15296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:41.600276 15296 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:38577
I20260811 03:09:41.600339 15432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:38577 every 8 connection(s)
I20260811 03:09:41.601646 15296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:09:41.605597 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 15296
I20260811 03:09:41.605811 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:09:41.606287 28084 external_mini_cluster.cc:1428] Setting key 35cb7783efe9701f0c5b90d513aff841
Aug 11 03:09:41 dist-test-slave-8tdl krb5kdc[14897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417781, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:41.625008 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.604670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:54777 (local address 127.27.109.62:32935)
0811 03:09:41.605464 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.605471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.606729 (+ 1258us) server_negotiation.cc:407] Connection header received
0811 03:09:41.609136 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.609142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.609279 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.609412 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:41.611317 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.612123 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.614042 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.614561 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.614660 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.618495 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:41.618500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:41.618504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:41.618540 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:41.621730 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:41.622762 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:41.622766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:41.622768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:41.622950 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:41.623374 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:41.623378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:41.623381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:41.623656 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:41.623774 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.624531 (+ 757us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.624712 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":465,"threads_started":1}
I20260811 03:09:41.626052 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.605450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:32935 (local address 127.27.109.3:54777)
0811 03:09:41.606472 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:41.606530 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:41.608861 (+ 2331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:41.609767 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:41.609777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:41.610455 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:41.611148 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.611166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.612297 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.612301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.613849 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.613862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.614865 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.614871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.615086 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.615793 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:41.615817 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:41.618233 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:41.621924 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:41.621936 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:41.621954 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:41.622604 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:41.623061 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:41.623065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:41.623067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:41.623266 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:41.623788 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:41.623794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:41.624153 (+ 359us) client_negotiation.cc:769] Sending connection context
0811 03:09:41.624410 (+ 257us) client_negotiation.cc:240] Negotiation successful
0811 03:09:41.624829 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":463,"threads_started":1}
I20260811 03:09:41.627786 15433 heartbeater.cc:347] Connected to a master server at 127.27.109.62:32935
I20260811 03:09:41.628090 15433 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:41.628834 15433 heartbeater.cc:511] Master 127.27.109.62:32935 requested a full tablet report, sending...
I20260811 03:09:41.630385 14944 ts_manager.cc:194] Registered new tserver with Master: 01fa4d2a02a94fcf8895ebc5e78f6482 (127.27.109.3:38577)
I20260811 03:09:41.631296 14944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:54777
I20260811 03:09:41.635468 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:09:41.655025 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.640329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.27.109.62:32935)
0811 03:09:41.640539 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.640549 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.641062 (+ 513us) server_negotiation.cc:407] Connection header received
0811 03:09:41.641164 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.641168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.641233 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.641321 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:41.642993 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.643892 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.645550 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.645965 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.646075 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.648849 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:41.648854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:41.648858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:41.648894 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:41.651943 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:41.652757 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:41.652761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:41.652764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:41.652915 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:41.653351 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:41.653355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:41.653357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:41.653713 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:41.653909 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.654353 (+ 444us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.654576 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 03:09:41.659826 14944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:09:41.667510 14944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:09:41.707195 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.681082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.27.109.1:37551)
0811 03:09:41.696387 (+ 15305us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.696399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.696442 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:09:41.696572 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.696582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.697022 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.697300 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:41.698603 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.700546 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.703690 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.705277 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.705412 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.705660 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.706521 (+ 861us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.706776 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15077,"thread_start_us":14892,"threads_started":1}
I20260811 03:09:41.707297 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.680610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37568)
0811 03:09:41.694822 (+ 14212us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:41.694925 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:41.695218 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:41.697684 (+ 2466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:41.697694 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:41.697738 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:41.698331 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.698352 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.700714 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.700720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.703462 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.703480 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.705899 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.705905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.706223 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.706292 (+ 69us) client_negotiation.cc:769] Sending connection context
0811 03:09:41.706443 (+ 151us) client_negotiation.cc:240] Negotiation successful
0811 03:09:41.706768 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13827,"mutex_wait_us":78,"thread_start_us":16955,"threads_started":1}
I20260811 03:09:41.707324 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.681104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.27.109.2:37303)
0811 03:09:41.693977 (+ 12873us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.693995 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.697558 (+ 3563us) server_negotiation.cc:407] Connection header received
0811 03:09:41.697996 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.698008 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.698294 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.698506 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:41.699811 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.701394 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.704534 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.705969 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.706123 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.706339 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.706829 (+ 490us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.707013 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12557,"thread_start_us":14354,"threads_started":1}
I20260811 03:09:41.707906 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.683916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38577 (local address 127.0.0.1:59186)
0811 03:09:41.697895 (+ 13979us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:41.697928 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:41.698086 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:41.699091 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:41.699096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:41.699125 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:41.699601 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.699612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.701342 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.701348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.703657 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.703669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.705876 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.705885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.706627 (+ 742us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.706657 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 03:09:41.706757 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 03:09:41.706887 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13831,"mutex_wait_us":549,"thread_start_us":13122,"threads_started":1}
I20260811 03:09:41.708238 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.680829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:37303 (local address 127.0.0.1:37876)
0811 03:09:41.697404 (+ 16575us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:41.697431 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:41.697700 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:41.698870 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:41.698876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:41.698908 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:41.699576 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.699596 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.701593 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.701599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.704349 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.704366 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.706285 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.706290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.706456 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.706483 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 03:09:41.706549 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 03:09:41.707654 (+ 1105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16407,"mutex_wait_us":33,"thread_start_us":13404,"threads_started":1}
I20260811 03:09:41.708626 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.684160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.27.109.3:38577)
0811 03:09:41.696816 (+ 12656us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.696825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.698240 (+ 1415us) server_negotiation.cc:407] Connection header received
0811 03:09:41.698417 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.698425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.698637 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.698907 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:41.699813 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.701159 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.703861 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.705277 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.705412 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.705660 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.707846 (+ 2186us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.708147 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12466,"thread_start_us":12271,"threads_started":1}
I20260811 03:09:41.711980 15219 tablet_service.cc:1467] Processing CreateTablet for tablet 702a613a0b084c919cdf8fea8e35b9a8 (DEFAULT_TABLE table=test-table [id=622ef878fb3f4d22a7a5423d05f6ac88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:41.712237 15076 tablet_service.cc:1467] Processing CreateTablet for tablet 702a613a0b084c919cdf8fea8e35b9a8 (DEFAULT_TABLE table=test-table [id=622ef878fb3f4d22a7a5423d05f6ac88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:41.712424 15363 tablet_service.cc:1467] Processing CreateTablet for tablet 702a613a0b084c919cdf8fea8e35b9a8 (DEFAULT_TABLE table=test-table [id=622ef878fb3f4d22a7a5423d05f6ac88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:41.713368 15219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702a613a0b084c919cdf8fea8e35b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:41.713869 15076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702a613a0b084c919cdf8fea8e35b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:41.713893 15363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702a613a0b084c919cdf8fea8e35b9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:41.730214 15454 tablet_bootstrap.cc:492] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Bootstrap starting.
I20260811 03:09:41.734483 15454 tablet_bootstrap.cc:654] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:41.734935 15455 tablet_bootstrap.cc:492] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767: Bootstrap starting.
I20260811 03:09:41.735699 15456 tablet_bootstrap.cc:492] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Bootstrap starting.
I20260811 03:09:41.736167 15454 log.cc:824] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:41.739212 15455 tablet_bootstrap.cc:654] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:41.739914 15456 tablet_bootstrap.cc:654] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:41.740239 15454 tablet_bootstrap.cc:492] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: No bootstrap required, opened a new log
I20260811 03:09:41.740586 15454 ts_tablet_manager.cc:1397] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260811 03:09:41.740670 15455 log.cc:824] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:41.741479 15456 log.cc:824] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:41.744040 15455 tablet_bootstrap.cc:492] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767: No bootstrap required, opened a new log
I20260811 03:09:41.744313 15455 ts_tablet_manager.cc:1397] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260811 03:09:41.745700 15456 tablet_bootstrap.cc:492] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: No bootstrap required, opened a new log
I20260811 03:09:41.745994 15456 ts_tablet_manager.cc:1397] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260811 03:09:41.752609 15455 raft_consensus.cc:348] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
I20260811 03:09:41.753067 15455 raft_consensus.cc:374] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:41.753168 15455 raft_consensus.cc:729] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a0d8cadf23844058e0f1ff941b46767, State: Initialized, Role: FOLLOWER
I20260811 03:09:41.753254 15454 raft_consensus.cc:348] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
I20260811 03:09:41.753794 15454 raft_consensus.cc:374] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:41.753929 15454 raft_consensus.cc:729] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401bda4c4e92419a9fd53db21051e2ef, State: Initialized, Role: FOLLOWER
I20260811 03:09:41.753914 15455 consensus_queue.cc:260] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
I20260811 03:09:41.754729 15454 consensus_queue.cc:260] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
I20260811 03:09:41.755919 15455 ts_tablet_manager.cc:1428] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260811 03:09:41.759694 15290 heartbeater.cc:503] Master 127.27.109.62:32935 was elected leader, sending a full tablet report...
I20260811 03:09:41.759904 15456 raft_consensus.cc:348] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
I20260811 03:09:41.760468 15456 raft_consensus.cc:374] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:41.760588 15456 raft_consensus.cc:729] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01fa4d2a02a94fcf8895ebc5e78f6482, State: Initialized, Role: FOLLOWER
I20260811 03:09:41.760617 15454 ts_tablet_manager.cc:1428] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260811 03:09:41.761389 15456 consensus_queue.cc:260] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
W20260811 03:09:41.761973 15291 tablet.cc:2369] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:09:41.762765 15433 heartbeater.cc:503] Master 127.27.109.62:32935 was elected leader, sending a full tablet report...
I20260811 03:09:41.763533 15456 ts_tablet_manager.cc:1428] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Time spent starting tablet: real 0.017s user 0.016s sys 0.003s
I20260811 03:09:41.795480 15462 raft_consensus.cc:482] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:09:41.795864 15462 raft_consensus.cc:504] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
I20260811 03:09:41.797818 15462 leader_election.cc:288] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a0d8cadf23844058e0f1ff941b46767 (127.27.109.1:37551), 01fa4d2a02a94fcf8895ebc5e78f6482 (127.27.109.3:38577)
I20260811 03:09:41.806525 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.798703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:41731 (local address 127.27.109.3:38577)
0811 03:09:41.799023 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.799029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.799786 (+ 757us) server_negotiation.cc:407] Connection header received
0811 03:09:41.800075 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.800083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.800180 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.800247 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:41.801255 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.802238 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.804345 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.805388 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.805530 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.805587 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.806046 (+ 459us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.806225 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260811 03:09:41.806563 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.798261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:38577 (local address 127.27.109.2:41731)
0811 03:09:41.799676 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:41.799692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:41.799865 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:41.800493 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:41.800500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:41.800585 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:41.801080 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.801096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.802404 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.802411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.804155 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.804168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.805652 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.805659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.805846 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.805884 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 03:09:41.805996 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 03:09:41.806153 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"spinlock_wait_cycles":43264,"thread_start_us":942,"threads_started":1}
I20260811 03:09:41.807364 15465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.798468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.27.109.2:57363)
0811 03:09:41.800333 (+ 1865us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:41.800354 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:41.800589 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:41.801196 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:41.801201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:41.801227 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:41.801615 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.801628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.803066 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.803071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.804768 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:41.804786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.806253 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.806259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:41.806459 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.806492 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 03:09:41.806610 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 03:09:41.806887 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1735,"spinlock_wait_cycles":40192,"thread_start_us":1524,"threads_started":1}
I20260811 03:09:41.807567 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.798560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:57363 (local address 127.27.109.1:37551)
0811 03:09:41.798825 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.798832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.800443 (+ 1611us) server_negotiation.cc:407] Connection header received
0811 03:09:41.800782 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.800787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.800871 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.800954 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:41.801801 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.802849 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.804995 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.806020 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.806167 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.806233 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.807099 (+ 866us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.807241 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 03:09:41.808019 15384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702a613a0b084c919cdf8fea8e35b9a8" candidate_uuid: "401bda4c4e92419a9fd53db21051e2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" is_pre_election: true
I20260811 03:09:41.808774 15384 raft_consensus.cc:2454] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401bda4c4e92419a9fd53db21051e2ef in term 0.
I20260811 03:09:41.808804 15097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702a613a0b084c919cdf8fea8e35b9a8" candidate_uuid: "401bda4c4e92419a9fd53db21051e2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a0d8cadf23844058e0f1ff941b46767" is_pre_election: true
I20260811 03:09:41.809475 15097 raft_consensus.cc:2454] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 401bda4c4e92419a9fd53db21051e2ef in term 0.
I20260811 03:09:41.809679 15173 leader_election.cc:302] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01fa4d2a02a94fcf8895ebc5e78f6482, 401bda4c4e92419a9fd53db21051e2ef; no voters:
I20260811 03:09:41.810206 15462 raft_consensus.cc:2781] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:09:41.810374 15462 raft_consensus.cc:482] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:09:41.810485 15462 raft_consensus.cc:3037] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:41.812915 15462 raft_consensus.cc:504] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
I20260811 03:09:41.813606 15462 leader_election.cc:288] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [CANDIDATE]: Term 1 election: Requested vote from peers 8a0d8cadf23844058e0f1ff941b46767 (127.27.109.1:37551), 01fa4d2a02a94fcf8895ebc5e78f6482 (127.27.109.3:38577)
I20260811 03:09:41.814208 15384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702a613a0b084c919cdf8fea8e35b9a8" candidate_uuid: "401bda4c4e92419a9fd53db21051e2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482"
I20260811 03:09:41.814236 15097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702a613a0b084c919cdf8fea8e35b9a8" candidate_uuid: "401bda4c4e92419a9fd53db21051e2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a0d8cadf23844058e0f1ff941b46767"
I20260811 03:09:41.814430 15384 raft_consensus.cc:3037] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:41.814448 15097 raft_consensus.cc:3037] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:41.816870 15097 raft_consensus.cc:2454] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401bda4c4e92419a9fd53db21051e2ef in term 1.
I20260811 03:09:41.816872 15384 raft_consensus.cc:2454] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401bda4c4e92419a9fd53db21051e2ef in term 1.
I20260811 03:09:41.817502 15173 leader_election.cc:302] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01fa4d2a02a94fcf8895ebc5e78f6482, 401bda4c4e92419a9fd53db21051e2ef; no voters:
I20260811 03:09:41.817847 15462 raft_consensus.cc:2781] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:41.818326 15462 raft_consensus.cc:686] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 1 LEADER]: Becoming Leader. State: Replica: 401bda4c4e92419a9fd53db21051e2ef, State: Running, Role: LEADER
I20260811 03:09:41.818897 15462 consensus_queue.cc:237] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } }
I20260811 03:09:41.824398 14944 catalog_manager.cc:5495] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 401bda4c4e92419a9fd53db21051e2ef (127.27.109.2). New cstate: current_term: 1 leader_uuid: "401bda4c4e92419a9fd53db21051e2ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401bda4c4e92419a9fd53db21051e2ef" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 37303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 } health_report { overall_health: UNKNOWN } } }
W20260811 03:09:41.853560 15434 tablet.cc:2369] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:09:41.870533 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.863862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.27.109.2:37303)
0811 03:09:41.864157 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.864162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.864232 (+ 70us) server_negotiation.cc:407] Connection header received
0811 03:09:41.864466 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.864471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.864580 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.864732 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:09:41.865325 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.866136 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.867711 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.868385 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.868522 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.869197 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:09:41.869796 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:09:41.869925 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.870090 (+ 165us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.870171 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 03:09:41.879709 15097 raft_consensus.cc:1261] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 1 FOLLOWER]: Refusing update from remote peer 401bda4c4e92419a9fd53db21051e2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:41.879712 15384 raft_consensus.cc:1261] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 1 FOLLOWER]: Refusing update from remote peer 401bda4c4e92419a9fd53db21051e2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:41.880946 15466 consensus_queue.cc:1035] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "01fa4d2a02a94fcf8895ebc5e78f6482" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:41.881469 15462 consensus_queue.cc:1035] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a0d8cadf23844058e0f1ff941b46767" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:41.914414 14943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:41.914796 14943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:41.918992 14943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61596738757c4d709c13aa41cef54769: Sending DeleteTablet for 3 replicas of tablet 702a613a0b084c919cdf8fea8e35b9a8
I20260811 03:09:41.920523 15363 tablet_service.cc:1514] Processing DeleteTablet for tablet 702a613a0b084c919cdf8fea8e35b9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:41 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:59186
I20260811 03:09:41.920516 15219 tablet_service.cc:1514] Processing DeleteTablet for tablet 702a613a0b084c919cdf8fea8e35b9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:41 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:37876
I20260811 03:09:41.921238 15076 tablet_service.cc:1514] Processing DeleteTablet for tablet 702a613a0b084c919cdf8fea8e35b9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:41 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:37568
I20260811 03:09:41.923929 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 15010
I20260811 03:09:41.957201 15480 tablet_replica.cc:330] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: stopping tablet replica
I20260811 03:09:41.957721 15482 tablet_replica.cc:330] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767: stopping tablet replica
I20260811 03:09:41.958099 15481 tablet_replica.cc:330] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: stopping tablet replica
I20260811 03:09:41.958467 15482 raft_consensus.cc:2229] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:41.959084 15480 raft_consensus.cc:2229] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:09:41.958598 15481 raft_consensus.cc:2229] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:41.959216 15482 raft_consensus.cc:2258] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:41.959784 15481 raft_consensus.cc:2258] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:41.960598 15480 raft_consensus.cc:2258] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:41.963357 15481 ts_tablet_manager.cc:1905] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:41.964481 15482 ts_tablet_manager.cc:1905] T 702a613a0b084c919cdf8fea8e35b9a8 P 8a0d8cadf23844058e0f1ff941b46767: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 03:09:41 dist-test-slave-8tdl krb5kdc[14897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:09:41.967140 15480 ts_tablet_manager.cc:1905] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:41.971287 15481 ts_tablet_manager.cc:1918] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:41.971540 15481 log.cc:1196] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/702a613a0b084c919cdf8fea8e35b9a8
I20260811 03:09:41.972221 15481 ts_tablet_manager.cc:1939] T 702a613a0b084c919cdf8fea8e35b9a8 P 01fa4d2a02a94fcf8895ebc5e78f6482: Deleting consensus metadata
I20260811 03:09:41.974591 14931 catalog_manager.cc:4841] TS 01fa4d2a02a94fcf8895ebc5e78f6482 (127.27.109.3:38577): tablet 702a613a0b084c919cdf8fea8e35b9a8 (table test-table [id=622ef878fb3f4d22a7a5423d05f6ac88]) successfully deleted
I20260811 03:09:41.976276 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:41.926266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37576 (local address 127.27.109.1:37551)
0811 03:09:41.926522 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:41.926531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:41.948358 (+ 21827us) server_negotiation.cc:407] Connection header received
0811 03:09:41.949141 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:41.949152 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:41.949287 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:41.949409 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:41.952836 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.954605 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.958579 (+ 3974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:41.959401 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:41.959544 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:41.966306 (+ 6762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:41.966325 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:41.966345 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:41.966406 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:41.971218 (+ 4812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:41.972655 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:41.972668 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:41.972672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:41.972892 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:41.973483 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:41.973489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:41.973492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:41.974028 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:41.974388 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:41.975503 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0811 03:09:41.975793 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 03:09:41.976558 15480 ts_tablet_manager.cc:1918] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:41.976886 15480 log.cc:1196] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/702a613a0b084c919cdf8fea8e35b9a8
I20260811 03:09:41.977361 15141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:41.977670 15480 ts_tablet_manager.cc:1939] T 702a613a0b084c919cdf8fea8e35b9a8 P 401bda4c4e92419a9fd53db21051e2ef: Deleting consensus metadata
I20260811 03:09:41.980440 14926 catalog_manager.cc:4841] TS 401bda4c4e92419a9fd53db21051e2ef (127.27.109.2:37303): tablet 702a613a0b084c919cdf8fea8e35b9a8 (table test-table [id=622ef878fb3f4d22a7a5423d05f6ac88]) successfully deleted
I20260811 03:09:42.108330 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 15010
W20260811 03:09:42.137326 14927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:37551 (error 108)
I20260811 03:09:42.138455 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 15152
I20260811 03:09:42.139864 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.139245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37582)
0811 03:09:42.139553 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:42.139685 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
W20260811 03:09:42.140199 14927 catalog_manager.cc:4584] TS 8a0d8cadf23844058e0f1ff941b46767 (127.27.109.1:37551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 702a613a0b084c919cdf8fea8e35b9a8: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Aug 11 03:09:42 dist-test-slave-8tdl krb5kdc[14897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:09:42.160303 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.139748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.27.109.2:37303)
0811 03:09:42.139964 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:42.139970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:42.139994 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:09:42.140177 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:42.140184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:42.140278 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:42.140405 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:42.142246 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:42.143780 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:42.145952 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:42.146504 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:42.146629 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:42.152928 (+ 6299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:42.152947 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:42.152964 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:42.153020 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:42.156576 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:42.157706 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:42.157716 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:42.157719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:42.157899 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:42.158429 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:42.158433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:42.158435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:42.158943 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:42.159359 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:42.159659 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 03:09:42.159922 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 03:09:42.161108 15284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:42.194175 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.193568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37588)
0811 03:09:42.193842 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:42.193961 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260811 03:09:42.195456 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.195087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37596)
0811 03:09:42.195236 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:42.195329 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 03:09:42.256408 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.255788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37610)
0811 03:09:42.256063 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:42.256191 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260811 03:09:42.257745 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.257260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37612)
0811 03:09:42.257485 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:42.257586 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 03:09:42.285395 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 15152
I20260811 03:09:42.311280 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 15296
Aug 11 03:09:42 dist-test-slave-8tdl krb5kdc[14897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:09:42.330176 15487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.312221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.27.109.3:38577)
0811 03:09:42.312761 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:42.312766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:42.312784 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:09:42.312842 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:42.312846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:42.312914 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:42.313016 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:42.314626 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:42.315566 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:42.317014 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:42.317415 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:42.317530 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:42.322348 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:42.322368 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:42.322386 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:42.322438 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:42.326013 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:42.327247 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:42.327256 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:42.327259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:42.327432 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:42.327970 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:42.327974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:42.327977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:42.328499 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:42.328770 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:42.329490 (+ 720us) server_negotiation.cc:299] Negotiation successful
0811 03:09:42.329706 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":303,"threads_started":1}
I20260811 03:09:42.331080 15428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:42.338351 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.337614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37616)
0811 03:09:42.337954 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:42.338085 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260811 03:09:42.339972 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.339380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37630)
0811 03:09:42.339661 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:42.339776 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260811 03:09:42.426501 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 15296
I20260811 03:09:42.449065 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 14910
I20260811 03:09:42.450425 14976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:42.569566 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:42.568984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37551 (local address 127.0.0.1:37638)
0811 03:09:42.569202 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:42.569328 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260811 03:09:42.569618 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 14910
2026-08-11T03:09:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4981 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-11T03:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:09:42Z Disabled control of system clock
I20260811 03:09:42.678119 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42161
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:42161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 03:09:43.072055 15493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:43.072402 15493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:43.072551 15493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:43.080924 15493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:09:43.081079 15493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:43.081118 15493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:43.081231 15493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:09:43.081295 15493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:09:43.091809 15493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:42161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:42161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:43.093345 15493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:43.095347 15493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:43.108385 15502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:43.108280 15499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:43.108319 15498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:43.109756 15493 server_base.cc:1034] running on GCE node
I20260811 03:09:43.110821 15493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:43.112488 15493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:43.113780 15493 hybrid_clock.cc:648] HybridClock initialized: now 1786417783113696 us; error 50 us; skew 500 ppm
I20260811 03:09:43.114286 15493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:43.117259 15493 webserver.cc:460] Webserver started at http://127.27.109.62:40959/ using document root <none> and password file <none>
I20260811 03:09:43.118116 15493 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:43.118249 15493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:43.118628 15493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:43.122185 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "a2f7eee713f2479eb530f26b2a21fa07"
format_stamp: "Formatted at 2026-08-11 03:09:43 on dist-test-slave-8tdl"
server_key: "4546771cb6c0629013df93e3a2d5d458"
server_key_iv: "8083c439b5ceab31eb78cd9a34462718"
server_key_version: "encryptionkey@0"
I20260811 03:09:43.123237 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "a2f7eee713f2479eb530f26b2a21fa07"
format_stamp: "Formatted at 2026-08-11 03:09:43 on dist-test-slave-8tdl"
server_key: "4546771cb6c0629013df93e3a2d5d458"
server_key_iv: "8083c439b5ceab31eb78cd9a34462718"
server_key_version: "encryptionkey@0"
I20260811 03:09:43.129447 15493 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:09:43.133766 15509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:43.135612 15493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260811 03:09:43.135857 15493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "a2f7eee713f2479eb530f26b2a21fa07"
format_stamp: "Formatted at 2026-08-11 03:09:43 on dist-test-slave-8tdl"
server_key: "4546771cb6c0629013df93e3a2d5d458"
server_key_iv: "8083c439b5ceab31eb78cd9a34462718"
server_key_version: "encryptionkey@0"
I20260811 03:09:43.136152 15493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:43.155676 15493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:43.156737 15493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:43.157096 15493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:43.179688 15493 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:42161
I20260811 03:09:43.179708 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:42161 every 8 connection(s)
I20260811 03:09:43.181314 15493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:09:43.185488 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 15493
I20260811 03:09:43.185638 15564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:43.185737 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:09:43.186261 28084 external_mini_cluster.cc:1428] Setting key 6f6c5d369cea48ba39f5b9c988fffe72
I20260811 03:09:43.200927 15564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07: Bootstrap starting.
I20260811 03:09:43.203500 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:43.188550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.27.109.62:42161)
0811 03:09:43.189943 (+ 1393us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:43.189956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:43.190074 (+ 118us) server_negotiation.cc:407] Connection header received
0811 03:09:43.192319 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:43.192346 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:43.192923 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:43.193312 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:43.194009 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:43.196072 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:43.198174 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:43.199113 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:43.199268 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:43.200659 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:43.200667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:43.200683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:43.201444 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:43.201538 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:43.201567 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:43.201658 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:43.201983 (+ 325us) server_negotiation.cc:299] Negotiation successful
0811 03:09:43.202373 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":586,"threads_started":1}
I20260811 03:09:43.204963 15564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:43.206287 15564 log.cc:824] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:43.209710 15564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07: No bootstrap required, opened a new log
I20260811 03:09:43.217860 15564 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f7eee713f2479eb530f26b2a21fa07" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42161 } }
I20260811 03:09:43.218327 15564 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:43.218457 15564 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f7eee713f2479eb530f26b2a21fa07, State: Initialized, Role: FOLLOWER
I20260811 03:09:43.219137 15564 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f7eee713f2479eb530f26b2a21fa07" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42161 } }
I20260811 03:09:43.219412 15564 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:09:43.219561 15564 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:09:43.219758 15564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:43.222116 15564 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f7eee713f2479eb530f26b2a21fa07" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42161 } }
I20260811 03:09:43.222741 15564 leader_election.cc:302] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2f7eee713f2479eb530f26b2a21fa07; no voters:
I20260811 03:09:43.223399 15564 leader_election.cc:288] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:09:43.223626 15569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:43.225217 15569 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [term 1 LEADER]: Becoming Leader. State: Replica: a2f7eee713f2479eb530f26b2a21fa07, State: Running, Role: LEADER
I20260811 03:09:43.225811 15569 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f7eee713f2479eb530f26b2a21fa07" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42161 } }
I20260811 03:09:43.225981 15564 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:09:43.229274 15571 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f7eee713f2479eb530f26b2a21fa07. Latest consensus state: current_term: 1 leader_uuid: "a2f7eee713f2479eb530f26b2a21fa07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f7eee713f2479eb530f26b2a21fa07" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42161 } } }
I20260811 03:09:43.229717 15571 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:43.229918 15570 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2f7eee713f2479eb530f26b2a21fa07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f7eee713f2479eb530f26b2a21fa07" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 42161 } } }
I20260811 03:09:43.230247 15570 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07 [sys.catalog]: This master's current role is: LEADER
W20260811 03:09:43.237964 15587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:09:43.238150 15587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:09:43.238353 15581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:09:43.240425 15581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:09:43.249610 15581 catalog_manager.cc:1347] Generated new cluster ID: 6d9ccd3a16e34d4d85efc82c579ae69f
I20260811 03:09:43.249758 15581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:09:43.269752 15581 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:09:43.271891 15581 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:09:43.282864 15581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07: Generated new TSK 0
I20260811 03:09:43.284075 15581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:09:43.297505 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42161
--builtin_ntp_servers=127.27.109.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 03:09:43.686731 15590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:43.687096 15590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:43.687191 15590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:43.695130 15590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:43.695245 15590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:43.695354 15590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:09:43.707847 15590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:43.709235 15590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:43.711256 15590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:43.728482 15596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:43.729131 15595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:43.729727 15598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:43.730450 15590 server_base.cc:1034] running on GCE node
I20260811 03:09:43.731271 15590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:43.732702 15590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:43.733971 15590 hybrid_clock.cc:648] HybridClock initialized: now 1786417783733898 us; error 66 us; skew 500 ppm
I20260811 03:09:43.734339 15590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:43.743300 15590 webserver.cc:460] Webserver started at http://127.27.109.1:32831/ using document root <none> and password file <none>
I20260811 03:09:43.744158 15590 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:43.744309 15590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:43.744661 15590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:43.747867 15590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "6262605453ac4991b75328ae88797a04"
format_stamp: "Formatted at 2026-08-11 03:09:43 on dist-test-slave-8tdl"
server_key: "6e37696b3980ab2b3da0160f942fc5f8"
server_key_iv: "cb33ea00235c94f809c1d165d31b16ed"
server_key_version: "encryptionkey@0"
I20260811 03:09:43.748793 15590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "6262605453ac4991b75328ae88797a04"
format_stamp: "Formatted at 2026-08-11 03:09:43 on dist-test-slave-8tdl"
server_key: "6e37696b3980ab2b3da0160f942fc5f8"
server_key_iv: "cb33ea00235c94f809c1d165d31b16ed"
server_key_version: "encryptionkey@0"
I20260811 03:09:43.754878 15590 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 03:09:43.758997 15606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:43.760622 15590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:09:43.760860 15590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "6262605453ac4991b75328ae88797a04"
format_stamp: "Formatted at 2026-08-11 03:09:43 on dist-test-slave-8tdl"
server_key: "6e37696b3980ab2b3da0160f942fc5f8"
server_key_iv: "cb33ea00235c94f809c1d165d31b16ed"
server_key_version: "encryptionkey@0"
I20260811 03:09:43.761125 15590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:43.791532 15590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:43.792567 15590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:43.792944 15590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:43.794180 15590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:43.796109 15590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:43.796289 15590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:43.796478 15590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:43.796593 15590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:43.835737 15590 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:35963
I20260811 03:09:43.835840 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:35963 every 8 connection(s)
I20260811 03:09:43.837158 15590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:09:43.845631 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 15590
I20260811 03:09:43.845824 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:09:43.846398 28084 external_mini_cluster.cc:1428] Setting key 441d434113aa8101178a3c25be05efd2
I20260811 03:09:43.849884 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42161
--builtin_ntp_servers=127.27.109.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 03:09:43.851984 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:43.839955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:58951 (local address 127.27.109.62:42161)
0811 03:09:43.840492 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:43.840496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:43.841734 (+ 1238us) server_negotiation.cc:407] Connection header received
0811 03:09:43.843671 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:43.843677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:43.843772 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:43.843906 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:43.845021 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:43.846161 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:43.848258 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:43.848897 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:43.849089 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:43.850665 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:43.850691 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:43.850699 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:43.850728 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:43.850792 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:43.850813 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:43.850926 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:43.851361 (+ 435us) server_negotiation.cc:299] Negotiation successful
0811 03:09:43.851491 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":292,"threads_started":1}
I20260811 03:09:43.853075 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:43.840538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42161 (local address 127.27.109.1:58951)
0811 03:09:43.841573 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:43.841618 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:43.843379 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:43.844161 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:43.844169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:43.844258 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:43.844834 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:43.844855 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:43.846378 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:43.846384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:43.848049 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:43.848065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:43.849160 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:43.849166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:43.849383 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:43.850281 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:09:43.850284 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:43.850339 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:09:43.850436 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:43.851084 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:43.851102 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:43.851109 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 03:09:43.851310 (+ 201us) client_negotiation.cc:240] Negotiation successful
0811 03:09:43.851660 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":287,"threads_started":1}
I20260811 03:09:43.855012 15724 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42161
I20260811 03:09:43.855381 15724 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:43.856397 15724 heartbeater.cc:511] Master 127.27.109.62:42161 requested a full tablet report, sending...
I20260811 03:09:43.858880 15527 ts_manager.cc:194] Registered new tserver with Master: 6262605453ac4991b75328ae88797a04 (127.27.109.1:35963)
I20260811 03:09:43.860409 15527 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.1:58951
W20260811 03:09:44.243606 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:44.243923 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:44.244071 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:44.251690 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:44.251813 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:44.252010 15728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:09:44.261667 15728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:44.263252 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:44.265396 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:44.280263 15736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:44.280339 15738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:44.280375 15735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:44.281315 15728 server_base.cc:1034] running on GCE node
I20260811 03:09:44.282022 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:44.283263 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:44.284505 15728 hybrid_clock.cc:648] HybridClock initialized: now 1786417784284448 us; error 48 us; skew 500 ppm
I20260811 03:09:44.284825 15728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:44.287703 15728 webserver.cc:460] Webserver started at http://127.27.109.2:45083/ using document root <none> and password file <none>
I20260811 03:09:44.288451 15728 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:44.288609 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:44.288975 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:44.292428 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "bf269520e2e842d9806be3e2ffaad7b8"
format_stamp: "Formatted at 2026-08-11 03:09:44 on dist-test-slave-8tdl"
server_key: "06ab69d1934282a560bcc89ef4b0daee"
server_key_iv: "3a5465e0fdcff03b6570080e5658e93b"
server_key_version: "encryptionkey@0"
I20260811 03:09:44.293563 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "bf269520e2e842d9806be3e2ffaad7b8"
format_stamp: "Formatted at 2026-08-11 03:09:44 on dist-test-slave-8tdl"
server_key: "06ab69d1934282a560bcc89ef4b0daee"
server_key_iv: "3a5465e0fdcff03b6570080e5658e93b"
server_key_version: "encryptionkey@0"
I20260811 03:09:44.299751 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260811 03:09:44.303844 15745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:44.305593 15728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:09:44.305806 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "bf269520e2e842d9806be3e2ffaad7b8"
format_stamp: "Formatted at 2026-08-11 03:09:44 on dist-test-slave-8tdl"
server_key: "06ab69d1934282a560bcc89ef4b0daee"
server_key_iv: "3a5465e0fdcff03b6570080e5658e93b"
server_key_version: "encryptionkey@0"
I20260811 03:09:44.306077 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:44.336247 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:44.337231 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:44.337597 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:44.338807 15728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:44.340497 15728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:44.340631 15728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:44.340785 15728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:44.340881 15728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:44.378928 15728 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:45331
I20260811 03:09:44.378968 15865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:45331 every 8 connection(s)
I20260811 03:09:44.380343 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:09:44.390560 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 15728
I20260811 03:09:44.390805 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:09:44.391368 28084 external_mini_cluster.cc:1428] Setting key 2c8143fbb968a88f4a96e2b4de9af0c4
I20260811 03:09:44.395090 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.383230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:39193 (local address 127.27.109.62:42161)
0811 03:09:44.383840 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:44.383847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:44.385070 (+ 1223us) server_negotiation.cc:407] Connection header received
0811 03:09:44.387040 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:44.387044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:44.387120 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:44.387224 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:44.388506 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.389387 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.391313 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.391943 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.392076 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:44.393962 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:44.393972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:44.393977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:44.394002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:44.394064 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:44.394084 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:44.394188 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:44.394458 (+ 270us) server_negotiation.cc:299] Negotiation successful
0811 03:09:44.394567 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":282,"threads_started":1}
I20260811 03:09:44.395139 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42161
--builtin_ntp_servers=127.27.109.20:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 03:09:44.396206 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.383830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42161 (local address 127.27.109.2:39193)
0811 03:09:44.384825 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:44.384882 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:44.386635 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:44.387515 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:44.387532 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:44.387638 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:44.388275 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:44.388294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.389571 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.389575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:44.391152 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:44.391167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.392253 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.392263 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:44.392518 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:44.393363 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:09:44.393366 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:44.393449 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:09:44.393592 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:44.394218 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:44.394229 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:44.394236 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 03:09:44.394409 (+ 173us) client_negotiation.cc:240] Negotiation successful
0811 03:09:44.394834 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":286,"threads_started":1}
I20260811 03:09:44.397950 15866 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42161
I20260811 03:09:44.398313 15866 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:44.399421 15866 heartbeater.cc:511] Master 127.27.109.62:42161 requested a full tablet report, sending...
I20260811 03:09:44.401338 15527 ts_manager.cc:194] Registered new tserver with Master: bf269520e2e842d9806be3e2ffaad7b8 (127.27.109.2:45331)
I20260811 03:09:44.402272 15527 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.2:39193
W20260811 03:09:44.760792 15871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:44.761132 15871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:44.761309 15871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:44.769119 15871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:44.769248 15871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:44.769425 15871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:09:44.779014 15871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:42161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:44.780405 15871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:44.782290 15871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:44.796540 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:44.796546 15879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:44.796876 15881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:44.797749 15871 server_base.cc:1034] running on GCE node
I20260811 03:09:44.798429 15871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:44.799531 15871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:44.800787 15871 hybrid_clock.cc:648] HybridClock initialized: now 1786417784800725 us; error 52 us; skew 500 ppm
I20260811 03:09:44.801153 15871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:44.809160 15871 webserver.cc:460] Webserver started at http://127.27.109.3:38963/ using document root <none> and password file <none>
I20260811 03:09:44.810115 15871 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:44.810285 15871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:44.810789 15871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:44.814503 15871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "614345a1620447f9891e697e573702b3"
format_stamp: "Formatted at 2026-08-11 03:09:44 on dist-test-slave-8tdl"
server_key: "ace4e4abac4bc3bf6f7023caecd46d29"
server_key_iv: "37bfca5327fb40290619dd360261df85"
server_key_version: "encryptionkey@0"
I20260811 03:09:44.815634 15871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "614345a1620447f9891e697e573702b3"
format_stamp: "Formatted at 2026-08-11 03:09:44 on dist-test-slave-8tdl"
server_key: "ace4e4abac4bc3bf6f7023caecd46d29"
server_key_iv: "37bfca5327fb40290619dd360261df85"
server_key_version: "encryptionkey@0"
I20260811 03:09:44.822098 15871 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:09:44.826540 15888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:44.828398 15871 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260811 03:09:44.828632 15871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "614345a1620447f9891e697e573702b3"
format_stamp: "Formatted at 2026-08-11 03:09:44 on dist-test-slave-8tdl"
server_key: "ace4e4abac4bc3bf6f7023caecd46d29"
server_key_iv: "37bfca5327fb40290619dd360261df85"
server_key_version: "encryptionkey@0"
I20260811 03:09:44.828890 15871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:44.864609 15724 heartbeater.cc:503] Master 127.27.109.62:42161 was elected leader, sending a full tablet report...
I20260811 03:09:44.877000 15871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:44.877913 15871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:44.878206 15871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:44.879354 15871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:44.881033 15871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:44.881132 15871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:44.881215 15871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:44.881273 15871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:44.920106 15871 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:39931
I20260811 03:09:44.920158 16002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:39931 every 8 connection(s)
I20260811 03:09:44.922000 15871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:09:44.923650 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 15871
I20260811 03:09:44.923861 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:09:44.924346 28084 external_mini_cluster.cc:1428] Setting key 86cece818661e995455a09e0c6fe4703
I20260811 03:09:44.936976 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.926080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:57413 (local address 127.27.109.62:42161)
0811 03:09:44.926577 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:44.926584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:44.927025 (+ 441us) server_negotiation.cc:407] Connection header received
0811 03:09:44.929479 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:44.929484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:44.929563 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:44.929661 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:44.931018 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.931864 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.933673 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.934221 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.934334 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:44.936063 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:44.936069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:44.936073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:44.936095 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:44.936147 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:44.936160 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:44.936247 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:44.936557 (+ 310us) server_negotiation.cc:299] Negotiation successful
0811 03:09:44.936655 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":1152,"threads_started":1}
I20260811 03:09:44.938081 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.925660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:42161 (local address 127.27.109.3:57413)
0811 03:09:44.926829 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:44.926888 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:44.929103 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:44.929975 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:44.929988 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:44.930120 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:44.930828 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:44.930860 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.932027 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.932032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:44.933495 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:44.933508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.934566 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.934577 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:44.934910 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:44.935672 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:09:44.935675 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:44.935720 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:09:44.935811 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:44.936288 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:44.936298 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:44.936306 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 03:09:44.936508 (+ 202us) client_negotiation.cc:240] Negotiation successful
0811 03:09:44.936882 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":543,"threads_started":1}
I20260811 03:09:44.939949 16003 heartbeater.cc:347] Connected to a master server at 127.27.109.62:42161
I20260811 03:09:44.940442 16003 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:44.941768 16003 heartbeater.cc:511] Master 127.27.109.62:42161 requested a full tablet report, sending...
I20260811 03:09:44.943729 15527 ts_manager.cc:194] Registered new tserver with Master: 614345a1620447f9891e697e573702b3 (127.27.109.3:39931)
I20260811 03:09:44.944629 15527 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.3:57413
I20260811 03:09:44.956619 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:09:44.968492 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.961723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.27.109.62:42161)
0811 03:09:44.961943 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:44.961950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:44.962558 (+ 608us) server_negotiation.cc:407] Connection header received
0811 03:09:44.962803 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:44.962807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:44.962889 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:44.962987 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:44.963544 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.964283 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.965721 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:44.966365 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:44.966478 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:44.967721 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:44.967726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:44.967729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:44.967751 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:44.967787 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:44.967799 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:44.967880 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:44.968140 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 03:09:44.968227 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260811 03:09:44.973063 15527 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:09:44.981103 15527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:09:45.009121 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.995172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39931 (local address 127.0.0.1:44002)
0811 03:09:44.996388 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:44.996464 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:44.996658 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:44.998100 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:44.998108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:44.998143 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:44.998632 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:44.998650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.002501 (+ 3851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.002509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.005601 (+ 3092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.005635 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.007962 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.007969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.008262 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.008346 (+ 84us) client_negotiation.cc:769] Sending connection context
0811 03:09:45.008489 (+ 143us) client_negotiation.cc:240] Negotiation successful
0811 03:09:45.008739 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":742,"threads_started":1}
I20260811 03:09:45.009666 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.995550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.27.109.3:39931)
0811 03:09:44.997372 (+ 1822us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:44.997383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:44.997437 (+ 54us) server_negotiation.cc:407] Connection header received
0811 03:09:44.997555 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:44.997563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:44.997847 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:44.998906 (+ 1059us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:44.999200 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.002222 (+ 3022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.005977 (+ 3755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.007685 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.008607 (+ 922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.008932 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.009025 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.009233 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1605,"thread_start_us":1424,"threads_started":1}
I20260811 03:09:45.014657 15937 tablet_service.cc:1467] Processing CreateTablet for tablet 348557669cb7451d867d617ce45993b3 (DEFAULT_TABLE table=test-table [id=7809a6fdb1514da8afa547071906e486]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:45.017311 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.997173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45331 (local address 127.0.0.1:52316)
0811 03:09:44.997745 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:44.997765 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:44.997885 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:45.008338 (+ 10453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:45.008344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:45.008380 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:45.008875 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.008890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.011804 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.011810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.014057 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.014073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.016384 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.016391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.016569 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.016607 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 03:09:45.016718 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 03:09:45.016875 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":26,"thread_start_us":318,"threads_started":1}
I20260811 03:09:45.017823 15937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348557669cb7451d867d617ce45993b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:45.018199 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.998192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34698)
0811 03:09:45.006888 (+ 8696us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.006907 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:45.007109 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:45.007755 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:45.007760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:45.007795 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:45.008383 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.008399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.011560 (+ 3161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.011566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.015277 (+ 3711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.015291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.017381 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.017388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.017580 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.017625 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 03:09:45.017715 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 03:09:45.017846 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8560,"mutex_wait_us":227,"thread_start_us":8329,"threads_started":1}
I20260811 03:09:45.018218 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.997532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.27.109.2:45331)
0811 03:09:45.006405 (+ 8873us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.006413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.006464 (+ 51us) server_negotiation.cc:407] Connection header received
0811 03:09:45.006594 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.006603 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.006956 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.009119 (+ 2163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:45.009806 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.011613 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.014268 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.016137 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.016314 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.016610 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.017017 (+ 407us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.017715 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8676,"thread_start_us":8541,"threads_started":1}
I20260811 03:09:45.018839 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:44.998376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34698 (local address 127.27.109.1:35963)
0811 03:09:45.006172 (+ 7796us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.006184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.007045 (+ 861us) server_negotiation.cc:407] Connection header received
0811 03:09:45.007308 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.007314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.007531 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.007768 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:45.008675 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.011393 (+ 2718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.015507 (+ 4114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.016811 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.016953 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.017229 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.018019 (+ 790us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.018240 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7489,"thread_start_us":7263,"threads_started":1}
I20260811 03:09:45.022536 15799 tablet_service.cc:1467] Processing CreateTablet for tablet 348557669cb7451d867d617ce45993b3 (DEFAULT_TABLE table=test-table [id=7809a6fdb1514da8afa547071906e486]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:45.022821 15658 tablet_service.cc:1467] Processing CreateTablet for tablet 348557669cb7451d867d617ce45993b3 (DEFAULT_TABLE table=test-table [id=7809a6fdb1514da8afa547071906e486]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:45.024178 15658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348557669cb7451d867d617ce45993b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:45.024189 15799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348557669cb7451d867d617ce45993b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:45.039636 16027 tablet_bootstrap.cc:492] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Bootstrap starting.
I20260811 03:09:45.041170 16026 tablet_bootstrap.cc:492] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04: Bootstrap starting.
I20260811 03:09:45.043545 16029 tablet_bootstrap.cc:492] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Bootstrap starting.
I20260811 03:09:45.045291 16026 tablet_bootstrap.cc:654] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:45.045585 16027 tablet_bootstrap.cc:654] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:45.046523 16026 log.cc:824] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:45.047032 16027 log.cc:824] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:45.047915 16029 tablet_bootstrap.cc:654] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:45.049584 16029 log.cc:824] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:45.056882 16027 tablet_bootstrap.cc:492] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: No bootstrap required, opened a new log
I20260811 03:09:45.056938 16026 tablet_bootstrap.cc:492] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04: No bootstrap required, opened a new log
I20260811 03:09:45.056938 16029 tablet_bootstrap.cc:492] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: No bootstrap required, opened a new log
I20260811 03:09:45.057247 16027 ts_tablet_manager.cc:1397] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.006s
I20260811 03:09:45.057256 16029 ts_tablet_manager.cc:1397] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20260811 03:09:45.057256 16026 ts_tablet_manager.cc:1397] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260811 03:09:45.066148 16027 raft_consensus.cc:348] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.066783 16027 raft_consensus.cc:374] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:45.066926 16027 raft_consensus.cc:729] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 614345a1620447f9891e697e573702b3, State: Initialized, Role: FOLLOWER
I20260811 03:09:45.067656 16027 consensus_queue.cc:260] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.068944 16003 heartbeater.cc:503] Master 127.27.109.62:42161 was elected leader, sending a full tablet report...
I20260811 03:09:45.068907 16026 raft_consensus.cc:348] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.068905 16029 raft_consensus.cc:348] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.069406 16029 raft_consensus.cc:374] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:45.069404 16026 raft_consensus.cc:374] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:45.069507 16029 raft_consensus.cc:729] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf269520e2e842d9806be3e2ffaad7b8, State: Initialized, Role: FOLLOWER
I20260811 03:09:45.069506 16026 raft_consensus.cc:729] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6262605453ac4991b75328ae88797a04, State: Initialized, Role: FOLLOWER
I20260811 03:09:45.069727 16027 ts_tablet_manager.cc:1428] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260811 03:09:45.070325 16026 consensus_queue.cc:260] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.070331 16029 consensus_queue.cc:260] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.074404 15866 heartbeater.cc:503] Master 127.27.109.62:42161 was elected leader, sending a full tablet report...
I20260811 03:09:45.074633 16029 ts_tablet_manager.cc:1428] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260811 03:09:45.074905 16026 ts_tablet_manager.cc:1428] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04: Time spent starting tablet: real 0.018s user 0.007s sys 0.013s
W20260811 03:09:45.089982 15725 tablet.cc:2369] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:09:45.132609 15867 tablet.cc:2369] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:09:45.174038 16004 tablet.cc:2369] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:09:45.221318 16034 raft_consensus.cc:482] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:09:45.221654 16034 raft_consensus.cc:504] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.222836 16036 raft_consensus.cc:482] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:09:45.223156 16036 raft_consensus.cc:504] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.223924 16034 leader_election.cc:288] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf269520e2e842d9806be3e2ffaad7b8 (127.27.109.2:45331), 6262605453ac4991b75328ae88797a04 (127.27.109.1:35963)
I20260811 03:09:45.225319 16036 leader_election.cc:288] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 614345a1620447f9891e697e573702b3 (127.27.109.3:39931), bf269520e2e842d9806be3e2ffaad7b8 (127.27.109.2:45331)
I20260811 03:09:45.232441 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.224326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45331 (local address 127.27.109.3:33355)
0811 03:09:45.224553 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.224578 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:45.224770 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:45.225427 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:45.225435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:45.225468 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:45.225702 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.225725 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.227257 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.227268 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.229646 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.229659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.231566 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.231574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.231797 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.231838 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 03:09:45.231949 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 03:09:45.232115 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":4}
I20260811 03:09:45.233314 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.224328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:33355 (local address 127.27.109.2:45331)
0811 03:09:45.224570 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.224577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.224677 (+ 100us) server_negotiation.cc:407] Connection header received
0811 03:09:45.225007 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.225012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.225108 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.225206 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:45.225964 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.227084 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.229868 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.231191 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.231374 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.231462 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.232813 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.233025 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260811 03:09:45.234726 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.225253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:37887 (local address 127.27.109.1:35963)
0811 03:09:45.225536 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.225545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.226271 (+ 726us) server_negotiation.cc:407] Connection header received
0811 03:09:45.226627 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.226633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.226775 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.226846 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:45.227932 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.229072 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.231759 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.233209 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.233350 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.233560 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.234269 (+ 709us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.234425 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":96}
I20260811 03:09:45.235193 15819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348557669cb7451d867d617ce45993b3" candidate_uuid: "614345a1620447f9891e697e573702b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf269520e2e842d9806be3e2ffaad7b8" is_pre_election: true
I20260811 03:09:45.235606 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.224881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.27.109.3:37887)
0811 03:09:45.226170 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.226189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:45.226420 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:45.227052 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:45.227056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:45.227081 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:45.227703 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.227723 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.229276 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.229285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.231532 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.231544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.233780 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.233786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.234052 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.234100 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 03:09:45.235011 (+ 911us) client_negotiation.cc:240] Negotiation successful
0811 03:09:45.235181 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":34,"thread_start_us":930,"threads_started":1}
I20260811 03:09:45.235903 15819 raft_consensus.cc:2454] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 614345a1620447f9891e697e573702b3 in term 0.
I20260811 03:09:45.236555 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.226220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:43133 (local address 127.27.109.3:39931)
0811 03:09:45.226502 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.226512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.227298 (+ 786us) server_negotiation.cc:407] Connection header received
0811 03:09:45.227616 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.227622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.227713 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.227786 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:45.228803 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.230274 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.233850 (+ 3576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.235263 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.235439 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.235513 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.236161 (+ 648us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.236301 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 03:09:45.236789 15892 leader_election.cc:302] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 614345a1620447f9891e697e573702b3, bf269520e2e842d9806be3e2ffaad7b8; no voters:
I20260811 03:09:45.237295 16034 raft_consensus.cc:2781] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:09:45.237462 16034 raft_consensus.cc:482] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:09:45.237195 15678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348557669cb7451d867d617ce45993b3" candidate_uuid: "614345a1620447f9891e697e573702b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6262605453ac4991b75328ae88797a04" is_pre_election: true
I20260811 03:09:45.237581 16034 raft_consensus.cc:3037] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:45.237777 15678 raft_consensus.cc:2454] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 614345a1620447f9891e697e573702b3 in term 0.
I20260811 03:09:45.238003 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.226031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39931 (local address 127.27.109.1:43133)
0811 03:09:45.227141 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.227216 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:45.227426 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:45.228005 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:45.228011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:45.228047 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:45.228612 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.228629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.230531 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.230539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.233600 (+ 3061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.233621 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.235738 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.235744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.235930 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.235976 (+ 46us) client_negotiation.cc:769] Sending connection context
0811 03:09:45.237112 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0811 03:09:45.237285 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":3,"thread_start_us":754,"threads_started":1}
I20260811 03:09:45.239941 15957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348557669cb7451d867d617ce45993b3" candidate_uuid: "6262605453ac4991b75328ae88797a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "614345a1620447f9891e697e573702b3" is_pre_election: true
I20260811 03:09:45.241158 16034 raft_consensus.cc:504] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.241707 15957 raft_consensus.cc:2379] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6262605453ac4991b75328ae88797a04 in current term 1: Already voted for candidate 614345a1620447f9891e697e573702b3 in this term.
I20260811 03:09:45.242364 16034 leader_election.cc:288] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [CANDIDATE]: Term 1 election: Requested vote from peers bf269520e2e842d9806be3e2ffaad7b8 (127.27.109.2:45331), 6262605453ac4991b75328ae88797a04 (127.27.109.1:35963)
I20260811 03:09:45.242830 15678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348557669cb7451d867d617ce45993b3" candidate_uuid: "614345a1620447f9891e697e573702b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6262605453ac4991b75328ae88797a04"
I20260811 03:09:45.242890 15819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348557669cb7451d867d617ce45993b3" candidate_uuid: "614345a1620447f9891e697e573702b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf269520e2e842d9806be3e2ffaad7b8"
I20260811 03:09:45.243081 15678 raft_consensus.cc:3037] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:45.243160 15819 raft_consensus.cc:3037] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:45.244827 16039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.227222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:45331 (local address 127.27.109.1:59691)
0811 03:09:45.236935 (+ 9713us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.236957 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:45.237159 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:45.237681 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:45.237686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:45.237713 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:45.238454 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.238469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.239862 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.239866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.241683 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:45.241706 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.243960 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.243967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:45.244158 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.244194 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 03:09:45.244299 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 03:09:45.244484 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9553,"thread_start_us":9514,"threads_started":1}
I20260811 03:09:45.244869 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.227730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:59691 (local address 127.27.109.2:45331)
0811 03:09:45.233896 (+ 6166us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.233902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.237043 (+ 3141us) server_negotiation.cc:407] Connection header received
0811 03:09:45.237326 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.237332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.237410 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.237480 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:45.238665 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.239633 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.242040 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.243524 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.243722 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.243803 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.244373 (+ 570us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.244536 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6059,"thread_start_us":7561,"threads_started":1}
I20260811 03:09:45.246917 15819 raft_consensus.cc:2454] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 614345a1620447f9891e697e573702b3 in term 1.
I20260811 03:09:45.247011 15678 raft_consensus.cc:2454] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 614345a1620447f9891e697e573702b3 in term 1.
I20260811 03:09:45.246969 15818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348557669cb7451d867d617ce45993b3" candidate_uuid: "6262605453ac4991b75328ae88797a04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf269520e2e842d9806be3e2ffaad7b8" is_pre_election: true
I20260811 03:09:45.247432 15818 raft_consensus.cc:2379] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6262605453ac4991b75328ae88797a04 in current term 1: Already voted for candidate 614345a1620447f9891e697e573702b3 in this term.
I20260811 03:09:45.247457 15892 leader_election.cc:302] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 614345a1620447f9891e697e573702b3, bf269520e2e842d9806be3e2ffaad7b8; no voters:
I20260811 03:09:45.247833 16034 raft_consensus.cc:2781] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:45.248171 15610 leader_election.cc:302] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6262605453ac4991b75328ae88797a04; no voters: 614345a1620447f9891e697e573702b3, bf269520e2e842d9806be3e2ffaad7b8
I20260811 03:09:45.248521 16034 raft_consensus.cc:686] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 1 LEADER]: Becoming Leader. State: Replica: 614345a1620447f9891e697e573702b3, State: Running, Role: LEADER
I20260811 03:09:45.248600 16036 raft_consensus.cc:2726] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 03:09:45.249254 16034 consensus_queue.cc:237] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } }
I20260811 03:09:45.253800 15526 catalog_manager.cc:5495] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 614345a1620447f9891e697e573702b3 (127.27.109.3). New cstate: current_term: 1 leader_uuid: "614345a1620447f9891e697e573702b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614345a1620447f9891e697e573702b3" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 } health_report { overall_health: UNKNOWN } } }
I20260811 03:09:45.312067 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.305753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.27.109.3:39931)
0811 03:09:45.305974 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.305979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.305999 (+ 20us) server_negotiation.cc:407] Connection header received
0811 03:09:45.306183 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.306187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.306295 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.306458 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:09:45.306990 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.307996 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.309576 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.310022 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.310124 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.310760 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:09:45.311361 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:09:45.311481 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.311611 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.311685 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:09:45.321913 15818 raft_consensus.cc:1261] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 1 FOLLOWER]: Refusing update from remote peer 614345a1620447f9891e697e573702b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:45.321934 15678 raft_consensus.cc:1261] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 1 FOLLOWER]: Refusing update from remote peer 614345a1620447f9891e697e573702b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:45.323117 16034 consensus_queue.cc:1035] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf269520e2e842d9806be3e2ffaad7b8" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 45331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:45.323489 16041 consensus_queue.cc:1035] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6262605453ac4991b75328ae88797a04" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:45.336217 16045 mvcc.cc:205] Tried to move back new op lower bound from 7317167248666128384 to 7317167248384581632. Current Snapshot: MvccSnapshot[applied={T|T < 7317167248666128384}]
I20260811 03:09:45.338457 16047 mvcc.cc:205] Tried to move back new op lower bound from 7317167248666128384 to 7317167248384581632. Current Snapshot: MvccSnapshot[applied={T|T < 7317167248666128384}]
I20260811 03:09:45.355937 15526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:45.356289 15526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:45.360615 15526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2f7eee713f2479eb530f26b2a21fa07: Sending DeleteTablet for 3 replicas of tablet 348557669cb7451d867d617ce45993b3
I20260811 03:09:45.361910 15937 tablet_service.cc:1514] Processing DeleteTablet for tablet 348557669cb7451d867d617ce45993b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:45 UTC) from {username='slave'} at 127.0.0.1:44002
I20260811 03:09:45.362140 15799 tablet_service.cc:1514] Processing DeleteTablet for tablet 348557669cb7451d867d617ce45993b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:45 UTC) from {username='slave'} at 127.0.0.1:52316
I20260811 03:09:45.362179 15658 tablet_service.cc:1514] Processing DeleteTablet for tablet 348557669cb7451d867d617ce45993b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:45 UTC) from {username='slave'} at 127.0.0.1:34698
I20260811 03:09:45.364297 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 15590
I20260811 03:09:45.374806 16056 tablet_replica.cc:330] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: stopping tablet replica
I20260811 03:09:45.375515 16056 raft_consensus.cc:2229] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:09:45.376654 16056 raft_consensus.cc:2258] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:45.379416 16057 tablet_replica.cc:330] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: stopping tablet replica
I20260811 03:09:45.379755 16056 ts_tablet_manager.cc:1905] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:45.380671 16057 raft_consensus.cc:2229] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:45.381328 16057 raft_consensus.cc:2258] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:45.385018 16058 tablet_replica.cc:330] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04: stopping tablet replica
I20260811 03:09:45.386068 16058 raft_consensus.cc:2229] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:45.388650 16056 ts_tablet_manager.cc:1918] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:45.388921 16056 log.cc:1196] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/348557669cb7451d867d617ce45993b3
I20260811 03:09:45.389670 16056 ts_tablet_manager.cc:1939] T 348557669cb7451d867d617ce45993b3 P 614345a1620447f9891e697e573702b3: Deleting consensus metadata
I20260811 03:09:45.390025 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.365554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.27.109.1:35963)
0811 03:09:45.365758 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.365765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.371646 (+ 5881us) server_negotiation.cc:407] Connection header received
0811 03:09:45.372874 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.372883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.373063 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.373163 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:45.374884 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.376046 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.382407 (+ 6361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.383148 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.383278 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.388040 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:45.388051 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:45.388068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:45.388910 (+ 842us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:45.388997 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:45.389031 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:45.389149 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.389476 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.389644 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 03:09:45.391384 15718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:45.392285 15514 catalog_manager.cc:4841] TS 614345a1620447f9891e697e573702b3 (127.27.109.3:39931): tablet 348557669cb7451d867d617ce45993b3 (table test-table [id=7809a6fdb1514da8afa547071906e486]) successfully deleted
I20260811 03:09:45.398039 16057 ts_tablet_manager.cc:1905] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:45.406816 16057 ts_tablet_manager.cc:1918] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:45.407244 16057 log.cc:1196] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/348557669cb7451d867d617ce45993b3
I20260811 03:09:45.407933 16057 ts_tablet_manager.cc:1939] T 348557669cb7451d867d617ce45993b3 P bf269520e2e842d9806be3e2ffaad7b8: Deleting consensus metadata
I20260811 03:09:45.410421 15514 catalog_manager.cc:4841] TS bf269520e2e842d9806be3e2ffaad7b8 (127.27.109.2:45331): tablet 348557669cb7451d867d617ce45993b3 (table test-table [id=7809a6fdb1514da8afa547071906e486]) successfully deleted
I20260811 03:09:45.493826 16058 raft_consensus.cc:2258] T 348557669cb7451d867d617ce45993b3 P 6262605453ac4991b75328ae88797a04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:45.494132 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 15590
W20260811 03:09:45.521164 15514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:35963 (error 108)
I20260811 03:09:45.522027 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 15728
I20260811 03:09:45.523494 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.522530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34704)
0811 03:09:45.523115 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.523259 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":339,"threads_started":1}
W20260811 03:09:45.523808 15514 catalog_manager.cc:4584] TS 6262605453ac4991b75328ae88797a04 (127.27.109.1:35963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 348557669cb7451d867d617ce45993b3: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
I20260811 03:09:45.530851 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.523135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.27.109.2:45331)
0811 03:09:45.523349 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.523356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.523438 (+ 82us) server_negotiation.cc:407] Connection header received
0811 03:09:45.523748 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.523753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.523883 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.524005 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:45.524521 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.525265 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.527437 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.527889 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.528003 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.529280 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:45.529289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:45.529304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:45.529982 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:45.530051 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:45.530074 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:45.530171 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.530339 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.530476 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:09:45.531572 15860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:45.577899 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.577332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34718)
0811 03:09:45.577591 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.577713 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260811 03:09:45.579269 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.578786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34722)
0811 03:09:45.578987 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.579094 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 03:09:45.628757 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 15728
I20260811 03:09:45.640108 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.639437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34738)
0811 03:09:45.639741 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.639874 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260811 03:09:45.641717 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.641104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34740)
0811 03:09:45.641392 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.641532 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 03:09:45.652262 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 15871
I20260811 03:09:45.661085 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.653110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.27.109.3:39931)
0811 03:09:45.653310 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:45.653317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:45.653347 (+ 30us) server_negotiation.cc:407] Connection header received
0811 03:09:45.653469 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:45.653475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:45.653608 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:45.653747 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:45.654210 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.655433 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.656821 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:45.657606 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:45.657751 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:45.659148 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:45.659161 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:45.659182 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:45.660139 (+ 957us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:45.660234 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:45.660271 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:45.660391 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:45.660577 (+ 186us) server_negotiation.cc:299] Negotiation successful
0811 03:09:45.660695 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:09:45.662010 15998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:45.722934 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.722150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34754)
0811 03:09:45.722544 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.722668 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260811 03:09:45.724578 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.723941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34762)
0811 03:09:45.724261 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.724360 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260811 03:09:45.798516 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 15871
I20260811 03:09:45.822862 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 15493
I20260811 03:09:45.824194 15559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:45.940618 16062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:45.939581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:35963 (local address 127.0.0.1:34776)
0811 03:09:45.940253 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:45.940384 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:35963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484}
I20260811 03:09:45.940623 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 15493
2026-08-11T03:09:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3367 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:09:46 dist-test-slave-8tdl krb5kdc[16068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:09:46 dist-test-slave-8tdl krb5kdc[16068](info): set up 2 sockets
Aug 11 03:09:46 dist-test-slave-8tdl krb5kdc[16068](info): commencing operation
krb5kdc: starting...
W20260811 03:09:47.154983 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.008s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:09:47 dist-test-slave-8tdl krb5kdc[16068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:09:47.508306 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:40075
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:47.883628 16081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:47.883944 16081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:47.884039 16081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:47.891682 16081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:09:47.891875 16081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:47.891952 16081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:47.892023 16081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:09:47.892072 16081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:09:47.903108 16081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:47.904546 16081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:47.905889 16081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:47.918533 16089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:47.918571 16086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:47.918620 16087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:47.924640 16081 server_base.cc:1034] running on GCE node
I20260811 03:09:47.925658 16081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:47.927443 16081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:47.928714 16081 hybrid_clock.cc:648] HybridClock initialized: now 1786417787928658 us; error 34 us; skew 500 ppm
I20260811 03:09:47.929148 16081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:47 dist-test-slave-8tdl krb5kdc[16068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417787, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:47.932751 16081 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:09:47.934603 16081 webserver.cc:460] Webserver started at http://127.27.109.62:43637/ using document root <none> and password file <none>
I20260811 03:09:47.935459 16081 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:47.935570 16081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:47.935842 16081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:47.939190 16081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "162a308b3dea4ec3aeef2d5e1a2019a6"
format_stamp: "Formatted at 2026-08-11 03:09:47 on dist-test-slave-8tdl"
server_key: "6c74e97d5e47bf1089a1fad3ac166f75"
server_key_iv: "ba668bc0955eec1b15c42643a6eee05f"
server_key_version: "encryptionkey@0"
I20260811 03:09:47.939983 16081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "162a308b3dea4ec3aeef2d5e1a2019a6"
format_stamp: "Formatted at 2026-08-11 03:09:47 on dist-test-slave-8tdl"
server_key: "6c74e97d5e47bf1089a1fad3ac166f75"
server_key_iv: "ba668bc0955eec1b15c42643a6eee05f"
server_key_version: "encryptionkey@0"
I20260811 03:09:47.945708 16081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 03:09:47.949676 16096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:47.951436 16081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:47.951660 16081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "162a308b3dea4ec3aeef2d5e1a2019a6"
format_stamp: "Formatted at 2026-08-11 03:09:47 on dist-test-slave-8tdl"
server_key: "6c74e97d5e47bf1089a1fad3ac166f75"
server_key_iv: "ba668bc0955eec1b15c42643a6eee05f"
server_key_version: "encryptionkey@0"
I20260811 03:09:47.951990 16081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:47.968416 16081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:47.973196 16081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:47.973532 16081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:47.994041 16081 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:40075
I20260811 03:09:47.994136 16150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:40075 every 8 connection(s)
I20260811 03:09:47.995599 16081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:09:48.000221 16151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:48.004469 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 16081
I20260811 03:09:48.004700 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:09:48.005280 28084 external_mini_cluster.cc:1428] Setting key 465ec357746d953aa38bd0f9863c455f
I20260811 03:09:48.015341 16151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6: Bootstrap starting.
I20260811 03:09:48.019769 16151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:48.021406 16151 log.cc:824] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6: Log is configured to *not* fsync() on all Append() calls
Aug 11 03:09:48 dist-test-slave-8tdl krb5kdc[16068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:48.025465 16151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6: No bootstrap required, opened a new log
I20260811 03:09:48.033043 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:48.007553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.27.109.62:40075)
0811 03:09:48.008501 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:48.008513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:48.008561 (+ 48us) server_negotiation.cc:407] Connection header received
0811 03:09:48.010401 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:48.010432 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:48.011006 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:48.011394 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:48.013448 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:48.015028 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:48.017239 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:48.018025 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:48.018175 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:48.024394 (+ 6219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:48.024403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:48.024421 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:48.024475 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:48.028133 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:48.029191 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:48.029203 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:48.029210 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:48.029397 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:48.029878 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:48.029881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:48.029883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:48.030547 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:48.030863 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:48.031347 (+ 484us) server_negotiation.cc:299] Negotiation successful
0811 03:09:48.031786 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":446,"threads_started":1}
I20260811 03:09:48.035485 16151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162a308b3dea4ec3aeef2d5e1a2019a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40075 } }
I20260811 03:09:48.035933 16151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:48.036058 16151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162a308b3dea4ec3aeef2d5e1a2019a6, State: Initialized, Role: FOLLOWER
I20260811 03:09:48.036746 16151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162a308b3dea4ec3aeef2d5e1a2019a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40075 } }
I20260811 03:09:48.037017 16151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:09:48.037158 16151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:09:48.037340 16151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:48.039705 16151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162a308b3dea4ec3aeef2d5e1a2019a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40075 } }
I20260811 03:09:48.040235 16151 leader_election.cc:302] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 162a308b3dea4ec3aeef2d5e1a2019a6; no voters:
I20260811 03:09:48.040932 16151 leader_election.cc:288] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:09:48.041168 16157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:48.041956 16157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [term 1 LEADER]: Becoming Leader. State: Replica: 162a308b3dea4ec3aeef2d5e1a2019a6, State: Running, Role: LEADER
I20260811 03:09:48.042605 16157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162a308b3dea4ec3aeef2d5e1a2019a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40075 } }
I20260811 03:09:48.043359 16151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:09:48.045920 16158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "162a308b3dea4ec3aeef2d5e1a2019a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162a308b3dea4ec3aeef2d5e1a2019a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40075 } } }
I20260811 03:09:48.046401 16158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:48.046052 16159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 162a308b3dea4ec3aeef2d5e1a2019a6. Latest consensus state: current_term: 1 leader_uuid: "162a308b3dea4ec3aeef2d5e1a2019a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "162a308b3dea4ec3aeef2d5e1a2019a6" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 40075 } } }
I20260811 03:09:48.046790 16159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6 [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:48.048453 16167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:09:48.054385 16167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:09:48.065062 16167 catalog_manager.cc:1347] Generated new cluster ID: 238dea5402b248cb989a4b384bf89c84
I20260811 03:09:48.065248 16167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:09:48.080080 16167 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:09:48.081511 16167 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:09:48.098582 16167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6: Generated new TSK 0
I20260811 03:09:48.099819 16167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:09:48.184670 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40075
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:48.551211 16182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:48.551533 16182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:48.551703 16182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:48.559876 16182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:48.560070 16182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:48.560254 16182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:09:48.572381 16182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:48.573907 16182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:48.575479 16182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:48.595350 16187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:48.595323 16190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:48.595381 16188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:48.596321 16182 server_base.cc:1034] running on GCE node
I20260811 03:09:48.597102 16182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:48.598274 16182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:48.599532 16182 hybrid_clock.cc:648] HybridClock initialized: now 1786417788599438 us; error 89 us; skew 500 ppm
I20260811 03:09:48.599891 16182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:48 dist-test-slave-8tdl krb5kdc[16068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417788, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:48.603654 16182 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:09:48.605515 16182 webserver.cc:460] Webserver started at http://127.27.109.1:40853/ using document root <none> and password file <none>
I20260811 03:09:48.606294 16182 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:48.606463 16182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:48.606889 16182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:48.610236 16182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "00268d79cf0f4b389fe5d2bcefc89526"
format_stamp: "Formatted at 2026-08-11 03:09:48 on dist-test-slave-8tdl"
server_key: "7d4542051623718cca61d043bc339684"
server_key_iv: "792c58a06b1fea6d3a286413c11100f4"
server_key_version: "encryptionkey@0"
I20260811 03:09:48.611164 16182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "00268d79cf0f4b389fe5d2bcefc89526"
format_stamp: "Formatted at 2026-08-11 03:09:48 on dist-test-slave-8tdl"
server_key: "7d4542051623718cca61d043bc339684"
server_key_iv: "792c58a06b1fea6d3a286413c11100f4"
server_key_version: "encryptionkey@0"
I20260811 03:09:48.617005 16182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 03:09:48.621194 16197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:48.622932 16182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:48.623152 16182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "00268d79cf0f4b389fe5d2bcefc89526"
format_stamp: "Formatted at 2026-08-11 03:09:48 on dist-test-slave-8tdl"
server_key: "7d4542051623718cca61d043bc339684"
server_key_iv: "792c58a06b1fea6d3a286413c11100f4"
server_key_version: "encryptionkey@0"
I20260811 03:09:48.623399 16182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:48.639611 16182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:48.644786 16182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:48.645140 16182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:48.646215 16182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:48.647955 16182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:48.648065 16182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:48.648221 16182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:48.648299 16182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:48.686077 16182 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:34975
I20260811 03:09:48.686195 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:34975 every 8 connection(s)
I20260811 03:09:48.687830 16182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:09:48.691452 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 16182
I20260811 03:09:48.691699 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:09:48.692231 28084 external_mini_cluster.cc:1428] Setting key 576f682f3c095ba6e04bfa699619bcae
Aug 11 03:09:48 dist-test-slave-8tdl krb5kdc[16068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417788, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:48.713964 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:48.690710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:39367 (local address 127.27.109.62:40075)
0811 03:09:48.691364 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:48.691372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:48.693599 (+ 2227us) server_negotiation.cc:407] Connection header received
0811 03:09:48.696113 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:48.696120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:48.696231 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:48.696381 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:48.698272 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:48.699244 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:48.701153 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:48.701635 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:48.701806 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:48.706110 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:48.706116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:48.706119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:48.706163 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:48.709718 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:48.710851 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:48.710862 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:48.710869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:48.711055 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:48.711572 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:48.711581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:48.711588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:48.712035 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:48.712228 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:48.713053 (+ 825us) server_negotiation.cc:299] Negotiation successful
0811 03:09:48.713356 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":337,"threads_started":1}
I20260811 03:09:48.714927 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:48.691357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:40075 (local address 127.27.109.1:39367)
0811 03:09:48.693389 (+ 2032us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:48.693454 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:48.695749 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:48.696697 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:48.696707 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:48.697352 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:48.698048 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:48.698070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:48.699462 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:48.699467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:48.700959 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:48.700976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:48.701965 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:48.701973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:48.702266 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:48.703151 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:48.703179 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:48.705781 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:48.710014 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:48.710029 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:48.710046 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:48.710610 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:48.711195 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:48.711199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:48.711202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:48.711429 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:48.712186 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:48.712193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:48.712589 (+ 396us) client_negotiation.cc:769] Sending connection context
0811 03:09:48.712858 (+ 269us) client_negotiation.cc:240] Negotiation successful
0811 03:09:48.713388 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1666,"thread_start_us":1474,"threads_started":1}
I20260811 03:09:48.716941 16315 heartbeater.cc:347] Connected to a master server at 127.27.109.62:40075
I20260811 03:09:48.717353 16315 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:48.718523 16315 heartbeater.cc:511] Master 127.27.109.62:40075 requested a full tablet report, sending...
I20260811 03:09:48.721577 16114 ts_manager.cc:194] Registered new tserver with Master: 00268d79cf0f4b389fe5d2bcefc89526 (127.27.109.1:34975)
I20260811 03:09:48.723320 16114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:39367
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:09:48.774386 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40075
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:49.141731 16321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:49.142035 16321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:49.142199 16321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:49.150133 16321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:49.150262 16321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:49.150449 16321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:09:49.161038 16321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:49.162629 16321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:49.164165 16321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:49.179220 16327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:49.179306 16328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:49.179306 16330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:49.179983 16321 server_base.cc:1034] running on GCE node
I20260811 03:09:49.180712 16321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:49.181808 16321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:49.183069 16321 hybrid_clock.cc:648] HybridClock initialized: now 1786417789182980 us; error 37 us; skew 500 ppm
I20260811 03:09:49.183435 16321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:49 dist-test-slave-8tdl krb5kdc[16068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:49.186883 16321 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:09:49.188822 16321 webserver.cc:460] Webserver started at http://127.27.109.2:34885/ using document root <none> and password file <none>
I20260811 03:09:49.189579 16321 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:49.189723 16321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:49.190089 16321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:49.193248 16321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "bd5af6f5549649848b236e7fec0501e7"
format_stamp: "Formatted at 2026-08-11 03:09:49 on dist-test-slave-8tdl"
server_key: "359e8d3edc63720ce5567a13b5125185"
server_key_iv: "684336da61c89200dea59715529ff5d3"
server_key_version: "encryptionkey@0"
I20260811 03:09:49.194085 16321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "bd5af6f5549649848b236e7fec0501e7"
format_stamp: "Formatted at 2026-08-11 03:09:49 on dist-test-slave-8tdl"
server_key: "359e8d3edc63720ce5567a13b5125185"
server_key_iv: "684336da61c89200dea59715529ff5d3"
server_key_version: "encryptionkey@0"
I20260811 03:09:49.200017 16321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260811 03:09:49.203893 16337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:49.205381 16321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260811 03:09:49.205621 16321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "bd5af6f5549649848b236e7fec0501e7"
format_stamp: "Formatted at 2026-08-11 03:09:49 on dist-test-slave-8tdl"
server_key: "359e8d3edc63720ce5567a13b5125185"
server_key_iv: "684336da61c89200dea59715529ff5d3"
server_key_version: "encryptionkey@0"
I20260811 03:09:49.205847 16321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:49.230983 16321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:49.236332 16321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:49.236698 16321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:49.237823 16321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:49.239526 16321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:49.239638 16321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:49.239807 16321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:49.239881 16321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:49.277555 16321 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:34365
I20260811 03:09:49.277612 16466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:34365 every 8 connection(s)
I20260811 03:09:49.278982 16321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:09:49.280539 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 16321
I20260811 03:09:49.280757 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:09:49.281240 28084 external_mini_cluster.cc:1428] Setting key 1fb4a714f6495826cf7c50399f387baf
Aug 11 03:09:49 dist-test-slave-8tdl krb5kdc[16068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:49.307916 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.282176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:51807 (local address 127.27.109.62:40075)
0811 03:09:49.283848 (+ 1672us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:49.283855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:49.287704 (+ 3849us) server_negotiation.cc:407] Connection header received
0811 03:09:49.289925 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:49.289932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:49.290040 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:49.290182 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:49.292092 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.292949 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.294832 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.295378 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.295540 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.300603 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:49.300609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:49.300613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:49.300652 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:49.303915 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:49.305087 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:49.305092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:49.305094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:49.305244 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:49.305804 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:49.305808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:49.305810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:49.306143 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:49.306272 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:49.307349 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0811 03:09:49.307579 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1537,"thread_start_us":1414,"threads_started":1}
I20260811 03:09:49.309543 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.282923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:40075 (local address 127.27.109.2:51807)
0811 03:09:49.287480 (+ 4557us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:49.287554 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:49.289589 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:49.290498 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:49.290508 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:49.291213 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:49.291882 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.291902 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.293138 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.293143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.294583 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.294597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.295661 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.295669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.295943 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.296743 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:49.296774 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:49.300284 (+ 3510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:49.304095 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:49.304107 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:49.304129 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:49.304887 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:49.305371 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:49.305379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:49.305386 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:49.305657 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:49.306298 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:49.306309 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:49.306839 (+ 530us) client_negotiation.cc:769] Sending connection context
0811 03:09:49.307183 (+ 344us) client_negotiation.cc:240] Negotiation successful
0811 03:09:49.307861 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4187,"thread_start_us":3877,"threads_started":1}
I20260811 03:09:49.311671 16467 heartbeater.cc:347] Connected to a master server at 127.27.109.62:40075
I20260811 03:09:49.312091 16467 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:49.312966 16467 heartbeater.cc:511] Master 127.27.109.62:40075 requested a full tablet report, sending...
I20260811 03:09:49.314879 16114 ts_manager.cc:194] Registered new tserver with Master: bd5af6f5549649848b236e7fec0501e7 (127.27.109.2:34365)
I20260811 03:09:49.316258 16114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:51807
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:09:49.363095 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40075
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:09:49.726970 16315 heartbeater.cc:503] Master 127.27.109.62:40075 was elected leader, sending a full tablet report...
W20260811 03:09:49.745800 16473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:49.746121 16473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:49.746287 16473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:49.754096 16473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:49.754226 16473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:49.754420 16473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:09:49.764762 16473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:49.766166 16473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:49.767664 16473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:49.781384 16478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:49.781445 16481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:49.781363 16479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:49.783066 16473 server_base.cc:1034] running on GCE node
I20260811 03:09:49.783739 16473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:49.784744 16473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:49.786002 16473 hybrid_clock.cc:648] HybridClock initialized: now 1786417789785915 us; error 44 us; skew 500 ppm
I20260811 03:09:49.786357 16473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:49 dist-test-slave-8tdl krb5kdc[16068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:49.789992 16473 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:09:49.791780 16473 webserver.cc:460] Webserver started at http://127.27.109.3:33005/ using document root <none> and password file <none>
I20260811 03:09:49.792546 16473 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:49.792685 16473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:49.793047 16473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:49.796284 16473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "214ce46cc8424488b42c9a9133c30c4c"
format_stamp: "Formatted at 2026-08-11 03:09:49 on dist-test-slave-8tdl"
server_key: "e97b0f0365e01b1bf0ed1fec9d6e6541"
server_key_iv: "148a0bf09af78dc3dc0be7844bc467c9"
server_key_version: "encryptionkey@0"
I20260811 03:09:49.797122 16473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "214ce46cc8424488b42c9a9133c30c4c"
format_stamp: "Formatted at 2026-08-11 03:09:49 on dist-test-slave-8tdl"
server_key: "e97b0f0365e01b1bf0ed1fec9d6e6541"
server_key_iv: "148a0bf09af78dc3dc0be7844bc467c9"
server_key_version: "encryptionkey@0"
I20260811 03:09:49.803143 16473 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260811 03:09:49.807073 16489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:49.808588 16473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:09:49.808779 16473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "214ce46cc8424488b42c9a9133c30c4c"
format_stamp: "Formatted at 2026-08-11 03:09:49 on dist-test-slave-8tdl"
server_key: "e97b0f0365e01b1bf0ed1fec9d6e6541"
server_key_iv: "148a0bf09af78dc3dc0be7844bc467c9"
server_key_version: "encryptionkey@0"
I20260811 03:09:49.808984 16473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:49.843079 16473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:49.848484 16473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:49.848816 16473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:49.850013 16473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:49.851635 16473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:49.851722 16473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:49.851872 16473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:49.851927 16473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:49.889690 16473 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:40847
I20260811 03:09:49.889787 16612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:40847 every 8 connection(s)
I20260811 03:09:49.891144 16473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:09:49.899727 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 16473
I20260811 03:09:49.899945 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:09:49.900424 28084 external_mini_cluster.cc:1428] Setting key c35125294fca3131dac735c6b7444f6b
Aug 11 03:09:49 dist-test-slave-8tdl krb5kdc[16068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:49.913838 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.894059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:45887 (local address 127.27.109.62:40075)
0811 03:09:49.894563 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:49.894568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:49.895652 (+ 1084us) server_negotiation.cc:407] Connection header received
0811 03:09:49.897623 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:49.897629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:49.897744 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:49.897877 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:49.899550 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.900489 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.902331 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.902845 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.903003 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.906820 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:49.906826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:49.906830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:49.906871 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:49.909776 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:49.910920 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:49.910925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:49.910927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:49.911133 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:49.911737 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:49.911741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:49.911744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:49.912121 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:49.912251 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:49.913109 (+ 858us) server_negotiation.cc:299] Negotiation successful
0811 03:09:49.913403 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":257,"threads_started":1}
I20260811 03:09:49.915200 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.894611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:40075 (local address 127.27.109.3:45887)
0811 03:09:49.895482 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:49.895522 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:49.897319 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:49.898129 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:49.898138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:49.898730 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:49.899370 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.899389 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.900680 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.900684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.902185 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.902204 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.903084 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.903094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.903366 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.904035 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:49.904058 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:49.906511 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:49.910028 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:49.910036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:49.910052 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:49.910779 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:49.911294 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:49.911298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:49.911301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:49.911596 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:49.912248 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:49.912254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:49.912650 (+ 396us) client_negotiation.cc:769] Sending connection context
0811 03:09:49.912986 (+ 336us) client_negotiation.cc:240] Negotiation successful
0811 03:09:49.913650 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":339,"threads_started":1}
I20260811 03:09:49.917130 16613 heartbeater.cc:347] Connected to a master server at 127.27.109.62:40075
I20260811 03:09:49.917454 16613 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:49.918223 16613 heartbeater.cc:511] Master 127.27.109.62:40075 requested a full tablet report, sending...
I20260811 03:09:49.919881 16114 ts_manager.cc:194] Registered new tserver with Master: 214ce46cc8424488b42c9a9133c30c4c (127.27.109.3:40847)
I20260811 03:09:49.920784 16114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:45887
I20260811 03:09:49.929832 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:09:49.951086 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.934662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.27.109.62:40075)
0811 03:09:49.935013 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:49.935022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:49.935381 (+ 359us) server_negotiation.cc:407] Connection header received
0811 03:09:49.935625 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:49.935635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:49.935701 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:49.935811 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:49.937365 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.938157 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.939675 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.940201 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.940355 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.944110 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:49.944116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:49.944120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:49.944163 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:49.947560 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:49.948601 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:49.948606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:49.948609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:49.948749 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:49.949308 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:49.949312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:49.949315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:49.949674 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:49.950017 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:49.950472 (+ 455us) server_negotiation.cc:299] Negotiation successful
0811 03:09:49.950742 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 03:09:49.955906 16114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:09:49.963428 16114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:09:49.990648 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.977402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:40847 (local address 127.0.0.1:35580)
0811 03:09:49.979019 (+ 1617us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:49.979062 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:49.979234 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:49.981536 (+ 2302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:49.981548 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:49.981594 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:49.982273 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.982291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.984745 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.984750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.987592 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.987612 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.989563 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.989572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.989777 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.989859 (+ 82us) client_negotiation.cc:769] Sending connection context
0811 03:09:49.989992 (+ 133us) client_negotiation.cc:240] Negotiation successful
0811 03:09:49.990198 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":30,"thread_start_us":2635,"threads_started":1}
I20260811 03:09:49.991688 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.981191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34975 (local address 127.0.0.1:43704)
0811 03:09:49.982418 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:49.982453 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:49.982596 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:49.983860 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:49.983865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:49.983893 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:49.984323 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.984334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.986791 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.986796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.988749 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.988762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.990829 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.990839 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.991017 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.991051 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:09:49.991140 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 03:09:49.991272 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":52,"thread_start_us":874,"threads_started":1}
I20260811 03:09:49.992138 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.980821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.27.109.1:34975)
0811 03:09:49.983229 (+ 2408us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:49.983239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:49.983272 (+ 33us) server_negotiation.cc:407] Connection header received
0811 03:09:49.983376 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:49.983383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:49.983640 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:49.983838 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:49.984821 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.986590 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.988932 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.990556 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.990743 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.991320 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:49.991399 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 03:09:49.991575 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2225,"thread_start_us":2075,"threads_started":1}
I20260811 03:09:49.992199 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.977735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.27.109.3:40847)
0811 03:09:49.980655 (+ 2920us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:49.980669 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:49.980718 (+ 49us) server_negotiation.cc:407] Connection header received
0811 03:09:49.980851 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:49.980863 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:49.981217 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:49.982550 (+ 1333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:49.982990 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.984563 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.987773 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.989321 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.991273 (+ 1952us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.991526 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:49.991597 (+ 71us) server_negotiation.cc:299] Negotiation successful
0811 03:09:49.991790 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2422,"thread_start_us":1862,"threads_started":1}
I20260811 03:09:49.992303 16626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.977902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:34365 (local address 127.0.0.1:36366)
0811 03:09:49.980336 (+ 2434us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:49.980357 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:49.980496 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:49.984214 (+ 3718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:49.984218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:49.984248 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:49.985412 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.985426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.987225 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.987232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.989480 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:49.989497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.991504 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.991512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:49.991690 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.991724 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:09:49.991795 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 03:09:49.991926 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2296,"mutex_wait_us":367,"thread_start_us":815,"threads_started":1}
I20260811 03:09:49.993280 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:49.978469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.27.109.2:34365)
0811 03:09:49.983561 (+ 5092us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:49.983569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:49.983598 (+ 29us) server_negotiation.cc:407] Connection header received
0811 03:09:49.983702 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:49.983709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:49.983938 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:49.984219 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:49.985811 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.987079 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.989676 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:49.991239 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:49.992544 (+ 1305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:49.992731 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:49.992794 (+ 63us) server_negotiation.cc:299] Negotiation successful
0811 03:09:49.992943 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4931,"thread_start_us":4793,"threads_started":1}
I20260811 03:09:49.996430 16544 tablet_service.cc:1467] Processing CreateTablet for tablet cf1db68b02c74f6b912c062973f59e8b (DEFAULT_TABLE table=test-table [id=0cc6b8303eaa47baa4a126667a52fbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:49.996441 16246 tablet_service.cc:1467] Processing CreateTablet for tablet cf1db68b02c74f6b912c062973f59e8b (DEFAULT_TABLE table=test-table [id=0cc6b8303eaa47baa4a126667a52fbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:49.998164 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1db68b02c74f6b912c062973f59e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:49.998164 16544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1db68b02c74f6b912c062973f59e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:50.002642 16393 tablet_service.cc:1467] Processing CreateTablet for tablet cf1db68b02c74f6b912c062973f59e8b (DEFAULT_TABLE table=test-table [id=0cc6b8303eaa47baa4a126667a52fbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:50.004317 16393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf1db68b02c74f6b912c062973f59e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:50.015869 16632 tablet_bootstrap.cc:492] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Bootstrap starting.
I20260811 03:09:50.018149 16633 tablet_bootstrap.cc:492] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Bootstrap starting.
I20260811 03:09:50.020009 16632 tablet_bootstrap.cc:654] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:50.021493 16632 log.cc:824] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:50.022138 16633 tablet_bootstrap.cc:654] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:50.023555 16633 log.cc:824] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:50.024541 16636 tablet_bootstrap.cc:492] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: Bootstrap starting.
I20260811 03:09:50.026216 16632 tablet_bootstrap.cc:492] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: No bootstrap required, opened a new log
I20260811 03:09:50.026556 16632 ts_tablet_manager.cc:1397] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260811 03:09:50.027841 16633 tablet_bootstrap.cc:492] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: No bootstrap required, opened a new log
I20260811 03:09:50.028160 16633 ts_tablet_manager.cc:1397] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260811 03:09:50.029091 16636 tablet_bootstrap.cc:654] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:50.030442 16636 log.cc:824] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:50.034109 16636 tablet_bootstrap.cc:492] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: No bootstrap required, opened a new log
I20260811 03:09:50.034425 16636 ts_tablet_manager.cc:1397] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260811 03:09:50.036887 16633 raft_consensus.cc:348] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.037318 16633 raft_consensus.cc:374] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:50.037420 16633 raft_consensus.cc:729] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00268d79cf0f4b389fe5d2bcefc89526, State: Initialized, Role: FOLLOWER
I20260811 03:09:50.037569 16632 raft_consensus.cc:348] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.038033 16632 raft_consensus.cc:374] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:50.038134 16632 raft_consensus.cc:729] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214ce46cc8424488b42c9a9133c30c4c, State: Initialized, Role: FOLLOWER
I20260811 03:09:50.038147 16633 consensus_queue.cc:260] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.038779 16632 consensus_queue.cc:260] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.040181 16613 heartbeater.cc:503] Master 127.27.109.62:40075 was elected leader, sending a full tablet report...
I20260811 03:09:50.040796 16632 ts_tablet_manager.cc:1428] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260811 03:09:50.043617 16633 ts_tablet_manager.cc:1428] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260811 03:09:50.045531 16636 raft_consensus.cc:348] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.046092 16636 raft_consensus.cc:374] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:50.046237 16636 raft_consensus.cc:729] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5af6f5549649848b236e7fec0501e7, State: Initialized, Role: FOLLOWER
I20260811 03:09:50.046958 16636 consensus_queue.cc:260] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.049299 16467 heartbeater.cc:503] Master 127.27.109.62:40075 was elected leader, sending a full tablet report...
I20260811 03:09:50.050092 16636 ts_tablet_manager.cc:1428] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260811 03:09:50.143025 16614 tablet.cc:2369] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:09:50.151654 16642 raft_consensus.cc:482] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:09:50.152267 16642 raft_consensus.cc:504] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.154593 16642 leader_election.cc:288] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214ce46cc8424488b42c9a9133c30c4c (127.27.109.3:40847), 00268d79cf0f4b389fe5d2bcefc89526 (127.27.109.1:34975)
I20260811 03:09:50.164374 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:50.156131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:58051 (local address 127.27.109.1:34975)
0811 03:09:50.156391 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:50.156397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:50.156719 (+ 322us) server_negotiation.cc:407] Connection header received
0811 03:09:50.156881 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:50.156885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:50.156958 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:50.157034 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:50.157612 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.158627 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.161765 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.163146 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.163328 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:50.163408 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:50.163965 (+ 557us) server_negotiation.cc:299] Negotiation successful
0811 03:09:50.164105 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 03:09:50.164371 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:50.155106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:45255 (local address 127.27.109.3:40847)
0811 03:09:50.155314 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:50.155319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:50.155888 (+ 569us) server_negotiation.cc:407] Connection header received
0811 03:09:50.156138 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:50.156144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:50.156239 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:50.156363 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:50.157189 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.158337 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.161765 (+ 3428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.163146 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.163327 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:50.163408 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:50.163965 (+ 557us) server_negotiation.cc:299] Negotiation successful
0811 03:09:50.164115 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 03:09:50.164623 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:50.155055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:40847 (local address 127.27.109.2:45255)
0811 03:09:50.155763 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:50.155783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:50.155944 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:50.156487 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:50.156494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:50.156522 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:50.156989 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:50.157001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.158503 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.158508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:50.161582 (+ 3074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:50.161591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.163483 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.163488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:50.163752 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:50.163808 (+ 56us) client_negotiation.cc:769] Sending connection context
0811 03:09:50.163915 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 03:09:50.164095 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":148,"thread_start_us":348,"threads_started":1}
I20260811 03:09:50.165064 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:50.155864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:34975 (local address 127.27.109.2:58051)
0811 03:09:50.156588 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:50.156607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:50.156737 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:50.157082 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:50.157090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:50.157122 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:50.157415 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:50.157423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.158803 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.158811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:50.161532 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:50.161555 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.163475 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.163482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:50.163758 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:50.163810 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 03:09:50.163915 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 03:09:50.164135 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":171,"thread_start_us":385,"threads_started":1}
I20260811 03:09:50.165853 16566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1db68b02c74f6b912c062973f59e8b" candidate_uuid: "bd5af6f5549649848b236e7fec0501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214ce46cc8424488b42c9a9133c30c4c" is_pre_election: true
I20260811 03:09:50.166152 16268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1db68b02c74f6b912c062973f59e8b" candidate_uuid: "bd5af6f5549649848b236e7fec0501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00268d79cf0f4b389fe5d2bcefc89526" is_pre_election: true
I20260811 03:09:50.166390 16566 raft_consensus.cc:2454] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5af6f5549649848b236e7fec0501e7 in term 0.
I20260811 03:09:50.166786 16268 raft_consensus.cc:2454] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd5af6f5549649848b236e7fec0501e7 in term 0.
I20260811 03:09:50.167287 16338 leader_election.cc:302] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214ce46cc8424488b42c9a9133c30c4c, bd5af6f5549649848b236e7fec0501e7; no voters:
I20260811 03:09:50.167747 16642 raft_consensus.cc:2781] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:09:50.167917 16642 raft_consensus.cc:482] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:09:50.168037 16642 raft_consensus.cc:3037] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:50.170734 16642 raft_consensus.cc:504] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.171480 16642 leader_election.cc:288] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [CANDIDATE]: Term 1 election: Requested vote from peers 214ce46cc8424488b42c9a9133c30c4c (127.27.109.3:40847), 00268d79cf0f4b389fe5d2bcefc89526 (127.27.109.1:34975)
I20260811 03:09:50.172200 16566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1db68b02c74f6b912c062973f59e8b" candidate_uuid: "bd5af6f5549649848b236e7fec0501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214ce46cc8424488b42c9a9133c30c4c"
I20260811 03:09:50.172367 16268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf1db68b02c74f6b912c062973f59e8b" candidate_uuid: "bd5af6f5549649848b236e7fec0501e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00268d79cf0f4b389fe5d2bcefc89526"
I20260811 03:09:50.172456 16566 raft_consensus.cc:3037] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:50.172662 16268 raft_consensus.cc:3037] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:50.175218 16566 raft_consensus.cc:2454] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5af6f5549649848b236e7fec0501e7 in term 1.
I20260811 03:09:50.175262 16268 raft_consensus.cc:2454] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd5af6f5549649848b236e7fec0501e7 in term 1.
I20260811 03:09:50.175792 16338 leader_election.cc:302] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00268d79cf0f4b389fe5d2bcefc89526, bd5af6f5549649848b236e7fec0501e7; no voters:
I20260811 03:09:50.176165 16642 raft_consensus.cc:2781] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:50.176755 16642 raft_consensus.cc:686] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 1 LEADER]: Becoming Leader. State: Replica: bd5af6f5549649848b236e7fec0501e7, State: Running, Role: LEADER
I20260811 03:09:50.177345 16642 consensus_queue.cc:237] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } }
I20260811 03:09:50.183156 16113 catalog_manager.cc:5495] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd5af6f5549649848b236e7fec0501e7 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "bd5af6f5549649848b236e7fec0501e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd5af6f5549649848b236e7fec0501e7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 34365 } health_report { overall_health: HEALTHY } } }
W20260811 03:09:50.190569 16316 tablet.cc:2369] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:09:50.238515 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:50.232319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.27.109.2:34365)
0811 03:09:50.232531 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:50.232536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:50.232585 (+ 49us) server_negotiation.cc:407] Connection header received
0811 03:09:50.232765 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:50.232769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:50.232846 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:50.232974 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:09:50.233359 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.234113 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.235772 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.236274 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.236385 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:50.237148 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:09:50.237684 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:09:50.237937 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:50.238208 (+ 271us) server_negotiation.cc:299] Negotiation successful
0811 03:09:50.238259 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 03:09:50.248301 16268 raft_consensus.cc:1261] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 1 FOLLOWER]: Refusing update from remote peer bd5af6f5549649848b236e7fec0501e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:50.248368 16566 raft_consensus.cc:1261] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 1 FOLLOWER]: Refusing update from remote peer bd5af6f5549649848b236e7fec0501e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:50.249480 16642 consensus_queue.cc:1035] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00268d79cf0f4b389fe5d2bcefc89526" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:50.249856 16645 consensus_queue.cc:1035] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214ce46cc8424488b42c9a9133c30c4c" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 40847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:50.264711 16651 mvcc.cc:205] Tried to move back new op lower bound from 7317167268843458560 to 7317167268572393472. Current Snapshot: MvccSnapshot[applied={T|T < 7317167268843458560}]
I20260811 03:09:50.271361 16650 mvcc.cc:205] Tried to move back new op lower bound from 7317167268843458560 to 7317167268572393472. Current Snapshot: MvccSnapshot[applied={T|T < 7317167268843458560}]
I20260811 03:09:50.284014 16113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:50.284356 16113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:50.289067 16113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 162a308b3dea4ec3aeef2d5e1a2019a6: Sending DeleteTablet for 3 replicas of tablet cf1db68b02c74f6b912c062973f59e8b
I20260811 03:09:50.290580 16544 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1db68b02c74f6b912c062973f59e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:50 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:35580
I20260811 03:09:50.290844 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1db68b02c74f6b912c062973f59e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:50 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:43704
I20260811 03:09:50.292075 16393 tablet_service.cc:1514] Processing DeleteTablet for tablet cf1db68b02c74f6b912c062973f59e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:50 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36366
I20260811 03:09:50.297005 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 16182
I20260811 03:09:50.320845 16661 tablet_replica.cc:330] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: stopping tablet replica
I20260811 03:09:50.321545 16661 raft_consensus.cc:2229] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:09:50.322536 16661 raft_consensus.cc:2258] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:50.323616 16660 tablet_replica.cc:330] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: stopping tablet replica
I20260811 03:09:50.324244 16660 raft_consensus.cc:2229] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:50.325039 16660 raft_consensus.cc:2258] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:50.325980 16658 tablet_replica.cc:330] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: stopping tablet replica
I20260811 03:09:50.326455 16658 raft_consensus.cc:2229] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:50.327023 16658 raft_consensus.cc:2258] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:50.328622 16660 ts_tablet_manager.cc:1905] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:50.328708 16661 ts_tablet_manager.cc:1905] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:50.330663 16658 ts_tablet_manager.cc:1905] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:50.337181 16660 ts_tablet_manager.cc:1918] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:50.337527 16660 log.cc:1196] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/cf1db68b02c74f6b912c062973f59e8b
I20260811 03:09:50.338294 16660 ts_tablet_manager.cc:1939] T cf1db68b02c74f6b912c062973f59e8b P 00268d79cf0f4b389fe5d2bcefc89526: Deleting consensus metadata
I20260811 03:09:50.339078 16661 ts_tablet_manager.cc:1918] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:50.339375 16661 log.cc:1196] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/cf1db68b02c74f6b912c062973f59e8b
I20260811 03:09:50.340030 16661 ts_tablet_manager.cc:1939] T cf1db68b02c74f6b912c062973f59e8b P bd5af6f5549649848b236e7fec0501e7: Deleting consensus metadata
I20260811 03:09:50.340858 16097 catalog_manager.cc:4841] TS 00268d79cf0f4b389fe5d2bcefc89526 (127.27.109.1:34975): tablet cf1db68b02c74f6b912c062973f59e8b (table test-table [id=0cc6b8303eaa47baa4a126667a52fbbd]) successfully deleted
Aug 11 03:09:50 dist-test-slave-8tdl krb5kdc[16068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:09:50.341926 16100 catalog_manager.cc:4841] TS bd5af6f5549649848b236e7fec0501e7 (127.27.109.2:34365): tablet cf1db68b02c74f6b912c062973f59e8b (table test-table [id=0cc6b8303eaa47baa4a126667a52fbbd]) successfully deleted
I20260811 03:09:50.343281 16658 ts_tablet_manager.cc:1918] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:50.343551 16658 log.cc:1196] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/cf1db68b02c74f6b912c062973f59e8b
I20260811 03:09:50.344188 16658 ts_tablet_manager.cc:1939] T cf1db68b02c74f6b912c062973f59e8b P 214ce46cc8424488b42c9a9133c30c4c: Deleting consensus metadata
I20260811 03:09:50.346124 16097 catalog_manager.cc:4841] TS 214ce46cc8424488b42c9a9133c30c4c (127.27.109.3:40847): tablet cf1db68b02c74f6b912c062973f59e8b (table test-table [id=0cc6b8303eaa47baa4a126667a52fbbd]) successfully deleted
I20260811 03:09:50.352195 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:50.298719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.27.109.1:34975)
0811 03:09:50.299067 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:50.299077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:50.319841 (+ 20764us) server_negotiation.cc:407] Connection header received
0811 03:09:50.320117 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:50.320124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:50.320244 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:50.320374 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:50.323973 (+ 3599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.325478 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.329336 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.330088 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.330215 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:50.343558 (+ 13343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:50.343624 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:50.343642 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:50.343693 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:50.348429 (+ 4736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:50.349499 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:50.349511 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:50.349515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:50.349698 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:50.350256 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:50.350263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:50.350269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:50.350826 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:50.351152 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:50.351536 (+ 384us) server_negotiation.cc:299] Negotiation successful
0811 03:09:50.351788 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260811 03:09:50.353103 16310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:50.449896 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 16182
I20260811 03:09:50.472527 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 16321
Aug 11 03:09:50 dist-test-slave-8tdl krb5kdc[16068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:09:50.490479 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:50.473545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.27.109.2:34365)
0811 03:09:50.473761 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:50.473767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:50.473789 (+ 22us) server_negotiation.cc:407] Connection header received
0811 03:09:50.473904 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:50.473914 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:50.473987 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:50.474125 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:50.475515 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.476403 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.477740 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.478196 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.478334 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:50.483449 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:50.483464 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:50.483480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:50.483524 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:50.487018 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:50.487989 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:50.487997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:50.487999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:50.488191 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:50.488626 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:50.488629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:50.488631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:50.489031 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:50.489495 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:50.489760 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 03:09:50.489987 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 03:09:50.491500 16462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:50.617645 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 16321
I20260811 03:09:50.642055 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 16473
Aug 11 03:09:50 dist-test-slave-8tdl krb5kdc[16068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:09:50.660580 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:50.643071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.27.109.3:40847)
0811 03:09:50.643271 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:50.643276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:50.643293 (+ 17us) server_negotiation.cc:407] Connection header received
0811 03:09:50.643353 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:50.643357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:50.643419 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:50.643533 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:50.645290 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.646221 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.648098 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:50.648499 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:50.648615 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:50.653852 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:50.653873 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:50.653890 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:50.653937 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:50.657214 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:50.658141 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:50.658152 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:50.658154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:50.658302 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:50.658781 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:50.658788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:50.658791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:50.659244 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:50.659503 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:50.659834 (+ 331us) server_negotiation.cc:299] Negotiation successful
0811 03:09:50.660119 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:09:50.661451 16607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:50.755908 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 16473
I20260811 03:09:50.778820 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 16081
I20260811 03:09:50.780076 16145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:50.896175 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 16081
2026-08-11T03:09:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4958 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-11T03:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:09:50Z Disabled control of system clock
I20260811 03:09:50.999775 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:44183
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:44183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 03:09:51.371610 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:51.371955 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:51.372089 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:51.379858 16670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:09:51.379997 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:51.380038 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:51.380155 16670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:09:51.380216 16670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:09:51.390645 16670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:44183
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:44183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:51.392352 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:51.394506 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:51.408631 16679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:51.408681 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:51.408704 16675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:51.409682 16670 server_base.cc:1034] running on GCE node
I20260811 03:09:51.410650 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:51.412401 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:51.413702 16670 hybrid_clock.cc:648] HybridClock initialized: now 1786417791413644 us; error 45 us; skew 500 ppm
I20260811 03:09:51.414160 16670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:51.416785 16670 webserver.cc:460] Webserver started at http://127.27.109.62:46529/ using document root <none> and password file <none>
I20260811 03:09:51.417578 16670 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:51.417729 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:51.418078 16670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:51.421334 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "b0b3734d33a943ff98440df577cd8ff5"
format_stamp: "Formatted at 2026-08-11 03:09:51 on dist-test-slave-8tdl"
server_key: "9776a23a952c542344092c99f86c1dbe"
server_key_iv: "02e0e3906d333379148560b1781c426a"
server_key_version: "encryptionkey@0"
I20260811 03:09:51.422223 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "b0b3734d33a943ff98440df577cd8ff5"
format_stamp: "Formatted at 2026-08-11 03:09:51 on dist-test-slave-8tdl"
server_key: "9776a23a952c542344092c99f86c1dbe"
server_key_iv: "02e0e3906d333379148560b1781c426a"
server_key_version: "encryptionkey@0"
I20260811 03:09:51.427927 16670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260811 03:09:51.432052 16687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:51.433696 16670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:09:51.433919 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "b0b3734d33a943ff98440df577cd8ff5"
format_stamp: "Formatted at 2026-08-11 03:09:51 on dist-test-slave-8tdl"
server_key: "9776a23a952c542344092c99f86c1dbe"
server_key_iv: "02e0e3906d333379148560b1781c426a"
server_key_version: "encryptionkey@0"
I20260811 03:09:51.434188 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:51.472124 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:51.473109 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:51.473456 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:51.493983 16670 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:44183
I20260811 03:09:51.494026 16742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:44183 every 8 connection(s)
I20260811 03:09:51.495545 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:09:51.496121 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 16670
I20260811 03:09:51.496320 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:09:51.497038 28084 external_mini_cluster.cc:1428] Setting key bd5c8810bf067e096e2306b3d2463794
I20260811 03:09:51.500510 16743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:51.511646 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:51.499336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.27.109.62:44183)
0811 03:09:51.500425 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:51.500436 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:51.500477 (+ 41us) server_negotiation.cc:407] Connection header received
0811 03:09:51.502311 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:51.502335 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:51.502938 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:51.503415 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:51.504022 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:51.505351 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:51.506962 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:51.507649 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:51.507858 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:51.509131 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:51.509139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:51.509153 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:51.509763 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:51.509844 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:51.509869 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:51.510022 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:51.510249 (+ 227us) server_negotiation.cc:299] Negotiation successful
0811 03:09:51.510605 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":455,"threads_started":1}
I20260811 03:09:51.512595 16743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5: Bootstrap starting.
I20260811 03:09:51.516166 16743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:51.517220 16743 log.cc:824] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:51.521128 16743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5: No bootstrap required, opened a new log
I20260811 03:09:51.529565 16743 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b3734d33a943ff98440df577cd8ff5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44183 } }
I20260811 03:09:51.530018 16743 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:51.530107 16743 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b3734d33a943ff98440df577cd8ff5, State: Initialized, Role: FOLLOWER
I20260811 03:09:51.530839 16743 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b3734d33a943ff98440df577cd8ff5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44183 } }
I20260811 03:09:51.531080 16743 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:09:51.531201 16743 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:09:51.531345 16743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:51.534065 16743 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b3734d33a943ff98440df577cd8ff5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44183 } }
I20260811 03:09:51.534621 16743 leader_election.cc:302] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0b3734d33a943ff98440df577cd8ff5; no voters:
I20260811 03:09:51.535384 16743 leader_election.cc:288] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:09:51.535607 16748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:51.536516 16748 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [term 1 LEADER]: Becoming Leader. State: Replica: b0b3734d33a943ff98440df577cd8ff5, State: Running, Role: LEADER
I20260811 03:09:51.537088 16748 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b3734d33a943ff98440df577cd8ff5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44183 } }
I20260811 03:09:51.537706 16743 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:09:51.540509 16750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0b3734d33a943ff98440df577cd8ff5. Latest consensus state: current_term: 1 leader_uuid: "b0b3734d33a943ff98440df577cd8ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b3734d33a943ff98440df577cd8ff5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44183 } } }
I20260811 03:09:51.541002 16750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:51.540792 16749 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0b3734d33a943ff98440df577cd8ff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b3734d33a943ff98440df577cd8ff5" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44183 } } }
I20260811 03:09:51.541455 16749 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5 [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:51.551532 16753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:09:51.557052 16753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:09:51.567955 16753 catalog_manager.cc:1347] Generated new cluster ID: 160fd19e04b846bcbd167070507ed65b
I20260811 03:09:51.568082 16753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:09:51.587780 16753 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:09:51.589217 16753 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:09:51.600870 16753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5: Generated new TSK 0
I20260811 03:09:51.601884 16753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:09:51.616634 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44183
--builtin_ntp_servers=127.27.109.20:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 03:09:52.009975 16769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:52.010324 16769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:52.010418 16769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:52.020294 16769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:52.020442 16769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:52.020658 16769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:09:52.031164 16769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:52.032560 16769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:52.034479 16769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:52.049683 16775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:52.049643 16778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:52.049607 16780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:52.050541 16769 server_base.cc:1034] running on GCE node
I20260811 03:09:52.051297 16769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:52.052419 16769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:52.053690 16769 hybrid_clock.cc:648] HybridClock initialized: now 1786417792053626 us; error 58 us; skew 500 ppm
I20260811 03:09:52.054124 16769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:52.056758 16769 webserver.cc:460] Webserver started at http://127.27.109.1:42999/ using document root <none> and password file <none>
I20260811 03:09:52.057582 16769 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:52.057730 16769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:52.058104 16769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:52.061318 16769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "d2acd637606a4c5ba2272d75564f3683"
format_stamp: "Formatted at 2026-08-11 03:09:52 on dist-test-slave-8tdl"
server_key: "413596a6fb034d10d6a0ff0da32a3aeb"
server_key_iv: "fcacef504c4b6dd3a711319996238dc5"
server_key_version: "encryptionkey@0"
I20260811 03:09:52.062325 16769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "d2acd637606a4c5ba2272d75564f3683"
format_stamp: "Formatted at 2026-08-11 03:09:52 on dist-test-slave-8tdl"
server_key: "413596a6fb034d10d6a0ff0da32a3aeb"
server_key_iv: "fcacef504c4b6dd3a711319996238dc5"
server_key_version: "encryptionkey@0"
I20260811 03:09:52.068267 16769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 03:09:52.072247 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:52.073884 16769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260811 03:09:52.074084 16769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "d2acd637606a4c5ba2272d75564f3683"
format_stamp: "Formatted at 2026-08-11 03:09:52 on dist-test-slave-8tdl"
server_key: "413596a6fb034d10d6a0ff0da32a3aeb"
server_key_iv: "fcacef504c4b6dd3a711319996238dc5"
server_key_version: "encryptionkey@0"
I20260811 03:09:52.074332 16769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:52.115303 16769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:52.116422 16769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:52.116811 16769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:52.118053 16769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:52.119719 16769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:52.119846 16769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:52.119977 16769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:52.120080 16769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:52.157802 16769 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37097
I20260811 03:09:52.157846 16905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37097 every 8 connection(s)
I20260811 03:09:52.159399 16769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:09:52.169747 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 16769
I20260811 03:09:52.170063 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:09:52.170568 28084 external_mini_cluster.cc:1428] Setting key 6b1fbc8cd129673afc8ad527890010c1
I20260811 03:09:52.174294 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44183
--builtin_ntp_servers=127.27.109.20:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 03:09:52.175196 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:52.162614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:54235 (local address 127.27.109.62:44183)
0811 03:09:52.163167 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:52.163172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:52.164366 (+ 1194us) server_negotiation.cc:407] Connection header received
0811 03:09:52.166961 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:52.166966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:52.167075 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:52.167203 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:52.168424 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:52.169184 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:52.171383 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:52.172125 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:52.172256 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:52.174097 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:52.174107 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:52.174112 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:52.174136 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:52.174192 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:52.174210 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:52.174325 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:52.174664 (+ 339us) server_negotiation.cc:299] Negotiation successful
0811 03:09:52.174801 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":336,"threads_started":1}
I20260811 03:09:52.176249 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:52.163162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44183 (local address 127.27.109.1:54235)
0811 03:09:52.164189 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:52.164230 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:52.166571 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:52.167481 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:52.167491 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:52.167595 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:52.168235 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:52.168252 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:52.169385 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:52.169391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:52.171150 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:52.171165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:52.172418 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:52.172426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:52.172830 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:52.173711 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:09:52.173714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:52.173763 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:09:52.173859 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:52.174321 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:52.174330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:52.174336 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 03:09:52.174549 (+ 213us) client_negotiation.cc:240] Negotiation successful
0811 03:09:52.175056 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":296,"threads_started":1}
I20260811 03:09:52.178326 16906 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44183
I20260811 03:09:52.178785 16906 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:52.179796 16906 heartbeater.cc:511] Master 127.27.109.62:44183 requested a full tablet report, sending...
I20260811 03:09:52.182394 16707 ts_manager.cc:194] Registered new tserver with Master: d2acd637606a4c5ba2272d75564f3683 (127.27.109.1:37097)
I20260811 03:09:52.184830 16707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.1:54235
W20260811 03:09:52.560861 16910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:52.561187 16910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:52.561277 16910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:52.569397 16910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:52.569530 16910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:52.569643 16910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:09:52.581355 16910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:52.582813 16910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:52.584751 16910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:52.599129 16922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:52.599291 16919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:52.599232 16920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:52.600529 16910 server_base.cc:1034] running on GCE node
I20260811 03:09:52.601251 16910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:52.602284 16910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:52.603539 16910 hybrid_clock.cc:648] HybridClock initialized: now 1786417792603465 us; error 65 us; skew 500 ppm
I20260811 03:09:52.603915 16910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:52.606516 16910 webserver.cc:460] Webserver started at http://127.27.109.2:35017/ using document root <none> and password file <none>
I20260811 03:09:52.607317 16910 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:52.607429 16910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:52.607787 16910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:52.610948 16910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6"
format_stamp: "Formatted at 2026-08-11 03:09:52 on dist-test-slave-8tdl"
server_key: "f9b1284c35e73ea31652cc01505c8235"
server_key_iv: "6fafeb8d2102af3930bcba51687fc821"
server_key_version: "encryptionkey@0"
I20260811 03:09:52.611842 16910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6"
format_stamp: "Formatted at 2026-08-11 03:09:52 on dist-test-slave-8tdl"
server_key: "f9b1284c35e73ea31652cc01505c8235"
server_key_iv: "6fafeb8d2102af3930bcba51687fc821"
server_key_version: "encryptionkey@0"
I20260811 03:09:52.618419 16910 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:09:52.622496 16929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:52.624089 16910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:52.624302 16910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6"
format_stamp: "Formatted at 2026-08-11 03:09:52 on dist-test-slave-8tdl"
server_key: "f9b1284c35e73ea31652cc01505c8235"
server_key_iv: "6fafeb8d2102af3930bcba51687fc821"
server_key_version: "encryptionkey@0"
I20260811 03:09:52.624563 16910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:52.637019 16910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:52.638034 16910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:52.638447 16910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:52.639796 16910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:52.641422 16910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:52.641562 16910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:52.641721 16910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:52.641813 16910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:52.678748 16910 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:39513
I20260811 03:09:52.678849 17046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:39513 every 8 connection(s)
I20260811 03:09:52.680296 16910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:09:52.682852 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 16910
I20260811 03:09:52.683172 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:09:52.683765 28084 external_mini_cluster.cc:1428] Setting key d39b02661fcd14893c78e62b7a76a81f
I20260811 03:09:52.687376 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44183
--builtin_ntp_servers=127.27.109.20:45947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 03:09:52.694924 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:52.683294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:44865 (local address 127.27.109.62:44183)
0811 03:09:52.683811 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:52.683816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:52.685478 (+ 1662us) server_negotiation.cc:407] Connection header received
0811 03:09:52.687805 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:52.687810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:52.687908 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:52.688029 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:52.689254 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:52.690016 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:52.691759 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:52.692191 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:52.692332 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:52.693879 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:52.693885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:52.693888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:52.693910 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:52.693957 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:52.693971 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:52.694075 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:52.694322 (+ 247us) server_negotiation.cc:299] Negotiation successful
0811 03:09:52.694445 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":284,"threads_started":1}
I20260811 03:09:52.696101 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:52.684043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44183 (local address 127.27.109.2:44865)
0811 03:09:52.685273 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:52.685331 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:52.687455 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:52.688309 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:52.688318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:52.688424 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:52.689069 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:52.689088 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:52.690188 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:52.690193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:52.691598 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:52.691612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:52.692446 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:52.692454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:52.692695 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:52.693501 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:09:52.693504 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:52.693564 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:09:52.693657 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:52.694075 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:52.694090 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:52.694100 (+ 10us) client_negotiation.cc:769] Sending connection context
0811 03:09:52.694315 (+ 215us) client_negotiation.cc:240] Negotiation successful
0811 03:09:52.694857 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":394,"threads_started":1}
I20260811 03:09:52.697919 17047 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44183
I20260811 03:09:52.698305 17047 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:52.699198 17047 heartbeater.cc:511] Master 127.27.109.62:44183 requested a full tablet report, sending...
I20260811 03:09:52.700958 16707 ts_manager.cc:194] Registered new tserver with Master: f1e12dfc3f4345cb8f4006bdd2083ea6 (127.27.109.2:39513)
I20260811 03:09:52.701860 16707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.2:44865
W20260811 03:09:53.061666 17051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:53.061991 17051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:53.062147 17051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:53.069798 17051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:53.069927 17051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:53.070112 17051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:09:53.080255 17051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:44183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:53.081691 17051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:53.083916 17051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:53.098384 17057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:53.098441 17058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:53.098465 17061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:53.099418 17051 server_base.cc:1034] running on GCE node
I20260811 03:09:53.100061 17051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:53.101085 17051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:53.102306 17051 hybrid_clock.cc:648] HybridClock initialized: now 1786417793102235 us; error 58 us; skew 500 ppm
I20260811 03:09:53.102722 17051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:53.105373 17051 webserver.cc:460] Webserver started at http://127.27.109.3:43955/ using document root <none> and password file <none>
I20260811 03:09:53.106088 17051 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:53.106177 17051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:53.106437 17051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:53.109570 17051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "f66225729d9140ed9d3dc7e927b224fb"
format_stamp: "Formatted at 2026-08-11 03:09:53 on dist-test-slave-8tdl"
server_key: "53b8ec60ffe7eddb9b1ec3b450ff77c3"
server_key_iv: "674ed05c675a0cef310b5c1d75b93a62"
server_key_version: "encryptionkey@0"
I20260811 03:09:53.110401 17051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "f66225729d9140ed9d3dc7e927b224fb"
format_stamp: "Formatted at 2026-08-11 03:09:53 on dist-test-slave-8tdl"
server_key: "53b8ec60ffe7eddb9b1ec3b450ff77c3"
server_key_iv: "674ed05c675a0cef310b5c1d75b93a62"
server_key_version: "encryptionkey@0"
I20260811 03:09:53.116550 17051 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260811 03:09:53.120469 17067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:53.122248 17051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 03:09:53.122454 17051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "f66225729d9140ed9d3dc7e927b224fb"
format_stamp: "Formatted at 2026-08-11 03:09:53 on dist-test-slave-8tdl"
server_key: "53b8ec60ffe7eddb9b1ec3b450ff77c3"
server_key_iv: "674ed05c675a0cef310b5c1d75b93a62"
server_key_version: "encryptionkey@0"
I20260811 03:09:53.122746 17051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:53.146669 17051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:53.147740 17051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:53.148078 17051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:53.149238 17051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:53.150961 17051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:53.151089 17051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:53.151224 17051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:53.151314 17051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:53.188769 16906 heartbeater.cc:503] Master 127.27.109.62:44183 was elected leader, sending a full tablet report...
I20260811 03:09:53.190142 17051 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:39133
I20260811 03:09:53.190148 17188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:39133 every 8 connection(s)
I20260811 03:09:53.191835 17051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:09:53.195070 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 17051
I20260811 03:09:53.195267 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:09:53.195735 28084 external_mini_cluster.cc:1428] Setting key 7992c64ad5cdc7f1b134e99e7ad55de9
I20260811 03:09:53.206833 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.194710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:45283 (local address 127.27.109.62:44183)
0811 03:09:53.194965 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.194972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.196799 (+ 1827us) server_negotiation.cc:407] Connection header received
0811 03:09:53.199433 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.199439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.199542 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.199639 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:53.200919 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.201714 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.203597 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.204044 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.204162 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.205853 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:53.205863 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:53.205871 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:53.205903 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:53.206023 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:53.206044 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:53.206153 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.206415 (+ 262us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.206511 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 03:09:53.208109 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.195398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:44183 (local address 127.27.109.3:45283)
0811 03:09:53.196612 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:53.196682 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:53.198998 (+ 2316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:53.199874 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:53.199884 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:53.199995 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:53.200709 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.200730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.201900 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.201904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.203407 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.203422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.204305 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.204310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.204548 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.205431 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:09:53.205434 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:53.205480 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:09:53.205575 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:53.206175 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:53.206185 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:53.206192 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 03:09:53.206369 (+ 177us) client_negotiation.cc:240] Negotiation successful
0811 03:09:53.206804 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":390,"threads_started":1}
I20260811 03:09:53.209996 17189 heartbeater.cc:347] Connected to a master server at 127.27.109.62:44183
I20260811 03:09:53.210428 17189 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:53.211542 17189 heartbeater.cc:511] Master 127.27.109.62:44183 requested a full tablet report, sending...
I20260811 03:09:53.213501 16707 ts_manager.cc:194] Registered new tserver with Master: f66225729d9140ed9d3dc7e927b224fb (127.27.109.3:39133)
I20260811 03:09:53.214541 16707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.3:45283
I20260811 03:09:53.225138 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:09:53.237345 17049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.230243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.27.109.62:44183)
0811 03:09:53.230457 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.230465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.230922 (+ 457us) server_negotiation.cc:407] Connection header received
0811 03:09:53.231165 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.231170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.231256 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.231383 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:53.231935 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.232718 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.234290 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.234833 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.234995 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.236469 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:53.236475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:53.236479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:53.236501 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:53.236545 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:53.236559 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:53.236720 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.236934 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.237002 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:09:53.242657 16707 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:09:53.251222 16707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:09:53.280128 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.266463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.27.109.2:39513)
0811 03:09:53.268165 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.268179 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.268911 (+ 732us) server_negotiation.cc:407] Connection header received
0811 03:09:53.269418 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.269429 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.269726 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.269893 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:53.271132 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.272757 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.275646 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.277264 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.277407 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.277713 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.279409 (+ 1696us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.279670 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1542,"thread_start_us":1425,"threads_started":1}
I20260811 03:09:53.281104 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.266366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39513 (local address 127.0.0.1:59674)
0811 03:09:53.268725 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:53.268780 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:53.269110 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:53.270263 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:53.270272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:53.270304 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:53.270899 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.270916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.272958 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.272963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.275413 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.275430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.277919 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.277925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.278246 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.278322 (+ 76us) client_negotiation.cc:769] Sending connection context
0811 03:09:53.280446 (+ 2124us) client_negotiation.cc:240] Negotiation successful
0811 03:09:53.280696 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2052,"mutex_wait_us":1,"thread_start_us":1884,"threads_started":1}
I20260811 03:09:53.282711 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.267161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:39133 (local address 127.0.0.1:50402)
0811 03:09:53.269272 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:53.269292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:53.269402 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:53.272069 (+ 2667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:53.272073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:53.272100 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:53.272500 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.272512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.276234 (+ 3722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.276240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.279092 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.279106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.281857 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.281864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.282034 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.282069 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 03:09:53.282162 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 03:09:53.282304 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1959,"mutex_wait_us":428,"thread_start_us":1807,"threads_started":1}
I20260811 03:09:53.282750 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.267929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.27.109.3:39133)
0811 03:09:53.268798 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.268808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.269547 (+ 739us) server_negotiation.cc:407] Connection header received
0811 03:09:53.269642 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.269648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.271572 (+ 1924us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.271772 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:53.272788 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.276037 (+ 3249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.279977 (+ 3940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.281648 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.281821 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.282145 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.282221 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.282394 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":525,"threads_started":1}
I20260811 03:09:53.285811 16979 tablet_service.cc:1467] Processing CreateTablet for tablet d65aeb1896a44d8084dc21df8141e134 (DEFAULT_TABLE table=test-table [id=d4e4dd692e334572b2d330013cf26fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:53.286425 17118 tablet_service.cc:1467] Processing CreateTablet for tablet d65aeb1896a44d8084dc21df8141e134 (DEFAULT_TABLE table=test-table [id=d4e4dd692e334572b2d330013cf26fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:53.287504 16979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65aeb1896a44d8084dc21df8141e134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:53.287925 17118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65aeb1896a44d8084dc21df8141e134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:53.288234 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.269711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.27.109.1:37097)
0811 03:09:53.279433 (+ 9722us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.279443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.279482 (+ 39us) server_negotiation.cc:407] Connection header received
0811 03:09:53.279635 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.279645 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.279977 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.280236 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:53.281069 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.282761 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.285336 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.286847 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.287021 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.287317 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.287659 (+ 342us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.287832 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9488,"thread_start_us":8581,"threads_started":1}
I20260811 03:09:53.288239 17207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.269287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:47120)
0811 03:09:53.279256 (+ 9969us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:53.279280 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:53.279421 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:53.280238 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:53.280243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:53.280270 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:53.280868 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.280880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.282961 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.282966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.285076 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.285098 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.287190 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.287200 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.287410 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.287459 (+ 49us) client_negotiation.cc:769] Sending connection context
0811 03:09:53.287577 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 03:09:53.287814 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9805,"thread_start_us":9599,"threads_started":1}
I20260811 03:09:53.292987 16836 tablet_service.cc:1467] Processing CreateTablet for tablet d65aeb1896a44d8084dc21df8141e134 (DEFAULT_TABLE table=test-table [id=d4e4dd692e334572b2d330013cf26fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:53.294770 16836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d65aeb1896a44d8084dc21df8141e134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:53.302376 17209 tablet_bootstrap.cc:492] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Bootstrap starting.
I20260811 03:09:53.304764 17210 tablet_bootstrap.cc:492] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Bootstrap starting.
I20260811 03:09:53.306886 17209 tablet_bootstrap.cc:654] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:53.308452 17209 log.cc:824] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:53.310503 17210 tablet_bootstrap.cc:654] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:53.311862 17212 tablet_bootstrap.cc:492] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: Bootstrap starting.
I20260811 03:09:53.312341 17210 log.cc:824] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:53.315778 17209 tablet_bootstrap.cc:492] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: No bootstrap required, opened a new log
I20260811 03:09:53.316113 17210 tablet_bootstrap.cc:492] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: No bootstrap required, opened a new log
I20260811 03:09:53.316131 17209 ts_tablet_manager.cc:1397] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260811 03:09:53.316195 17212 tablet_bootstrap.cc:654] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:53.316411 17210 ts_tablet_manager.cc:1397] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260811 03:09:53.317642 17212 log.cc:824] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:53.321686 17212 tablet_bootstrap.cc:492] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: No bootstrap required, opened a new log
I20260811 03:09:53.322052 17212 ts_tablet_manager.cc:1397] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260811 03:09:53.325488 17210 raft_consensus.cc:348] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.325960 17210 raft_consensus.cc:374] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:53.326067 17210 raft_consensus.cc:729] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e12dfc3f4345cb8f4006bdd2083ea6, State: Initialized, Role: FOLLOWER
I20260811 03:09:53.327121 17210 consensus_queue.cc:260] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.328673 17047 heartbeater.cc:503] Master 127.27.109.62:44183 was elected leader, sending a full tablet report...
I20260811 03:09:53.328675 17209 raft_consensus.cc:348] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.329293 17209 raft_consensus.cc:374] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:53.329396 17209 raft_consensus.cc:729] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66225729d9140ed9d3dc7e927b224fb, State: Initialized, Role: FOLLOWER
I20260811 03:09:53.329433 17210 ts_tablet_manager.cc:1428] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260811 03:09:53.330190 17209 consensus_queue.cc:260] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.331619 17189 heartbeater.cc:503] Master 127.27.109.62:44183 was elected leader, sending a full tablet report...
I20260811 03:09:53.332356 17209 ts_tablet_manager.cc:1428] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260811 03:09:53.333621 17212 raft_consensus.cc:348] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.334161 17212 raft_consensus.cc:374] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:53.334290 17212 raft_consensus.cc:729] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2acd637606a4c5ba2272d75564f3683, State: Initialized, Role: FOLLOWER
I20260811 03:09:53.335224 17212 consensus_queue.cc:260] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.337400 17212 ts_tablet_manager.cc:1428] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: Time spent starting tablet: real 0.015s user 0.011s sys 0.006s
W20260811 03:09:53.412494 16907 tablet.cc:2369] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:09:53.432821 17048 tablet.cc:2369] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:09:53.443738 17190 tablet.cc:2369] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:09:53.670264 17216 raft_consensus.cc:482] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:09:53.670823 17216 raft_consensus.cc:504] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.672816 17216 leader_election.cc:288] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1e12dfc3f4345cb8f4006bdd2083ea6 (127.27.109.2:39513), d2acd637606a4c5ba2272d75564f3683 (127.27.109.1:37097)
I20260811 03:09:53.679960 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.673207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:39513 (local address 127.27.109.3:37991)
0811 03:09:53.673472 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:53.673497 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:53.673657 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:53.674229 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:53.674238 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:53.674279 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:53.674445 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.674453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.675665 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.675671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.677240 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.677254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.679080 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.679087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.679287 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.679375 (+ 88us) client_negotiation.cc:769] Sending connection context
0811 03:09:53.679472 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 03:09:53.679635 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260811 03:09:53.680006 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.673315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:37991 (local address 127.27.109.2:39513)
0811 03:09:53.673593 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.673600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.673626 (+ 26us) server_negotiation.cc:407] Connection header received
0811 03:09:53.673835 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.673843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.673971 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.674090 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:53.674641 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.675516 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.677428 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.678861 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.679052 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.679138 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.679549 (+ 411us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.679711 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260811 03:09:53.680809 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.674058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:57047 (local address 127.27.109.1:37097)
0811 03:09:53.674279 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.674288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.674312 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:09:53.674590 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.674595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.674719 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.674852 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:53.675480 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.676197 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.678170 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.679566 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.679761 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.679835 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.680449 (+ 614us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.680628 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 03:09:53.681478 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.673668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.27.109.3:57047)
0811 03:09:53.674238 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:53.674257 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:53.674419 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:53.674867 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:53.674942 (+ 75us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:53.674968 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:53.675306 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.675315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.676375 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.676381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.677947 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:53.677963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.679957 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.679964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:53.680257 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.680291 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:09:53.681026 (+ 735us) client_negotiation.cc:240] Negotiation successful
0811 03:09:53.681174 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":278,"threads_started":1}
I20260811 03:09:53.681529 17000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65aeb1896a44d8084dc21df8141e134" candidate_uuid: "f66225729d9140ed9d3dc7e927b224fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" is_pre_election: true
I20260811 03:09:53.682201 17000 raft_consensus.cc:2454] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66225729d9140ed9d3dc7e927b224fb in term 0.
I20260811 03:09:53.682847 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65aeb1896a44d8084dc21df8141e134" candidate_uuid: "f66225729d9140ed9d3dc7e927b224fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2acd637606a4c5ba2272d75564f3683" is_pre_election: true
I20260811 03:09:53.683094 17069 leader_election.cc:302] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e12dfc3f4345cb8f4006bdd2083ea6, f66225729d9140ed9d3dc7e927b224fb; no voters:
I20260811 03:09:53.683478 16860 raft_consensus.cc:2454] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f66225729d9140ed9d3dc7e927b224fb in term 0.
I20260811 03:09:53.683586 17216 raft_consensus.cc:2781] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:09:53.683810 17216 raft_consensus.cc:482] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:09:53.683940 17216 raft_consensus.cc:3037] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:53.687317 17216 raft_consensus.cc:504] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.688180 17216 leader_election.cc:288] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [CANDIDATE]: Term 1 election: Requested vote from peers f1e12dfc3f4345cb8f4006bdd2083ea6 (127.27.109.2:39513), d2acd637606a4c5ba2272d75564f3683 (127.27.109.1:37097)
I20260811 03:09:53.688902 17000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65aeb1896a44d8084dc21df8141e134" candidate_uuid: "f66225729d9140ed9d3dc7e927b224fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6"
I20260811 03:09:53.688930 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d65aeb1896a44d8084dc21df8141e134" candidate_uuid: "f66225729d9140ed9d3dc7e927b224fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2acd637606a4c5ba2272d75564f3683"
I20260811 03:09:53.689178 16860 raft_consensus.cc:3037] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:53.689231 17000 raft_consensus.cc:3037] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:53.692855 17000 raft_consensus.cc:2454] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66225729d9140ed9d3dc7e927b224fb in term 1.
I20260811 03:09:53.692850 16860 raft_consensus.cc:2454] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f66225729d9140ed9d3dc7e927b224fb in term 1.
I20260811 03:09:53.693467 17069 leader_election.cc:302] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f1e12dfc3f4345cb8f4006bdd2083ea6, f66225729d9140ed9d3dc7e927b224fb; no voters:
I20260811 03:09:53.693933 17216 raft_consensus.cc:2781] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:53.694481 17216 raft_consensus.cc:686] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 1 LEADER]: Becoming Leader. State: Replica: f66225729d9140ed9d3dc7e927b224fb, State: Running, Role: LEADER
I20260811 03:09:53.695122 17216 consensus_queue.cc:237] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } }
I20260811 03:09:53.700805 16706 catalog_manager.cc:5495] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb reported cstate change: term changed from 0 to 1, leader changed from <none> to f66225729d9140ed9d3dc7e927b224fb (127.27.109.3). New cstate: current_term: 1 leader_uuid: "f66225729d9140ed9d3dc7e927b224fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f66225729d9140ed9d3dc7e927b224fb" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 39133 } health_report { overall_health: HEALTHY } } }
I20260811 03:09:53.746353 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.739275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.27.109.3:39133)
0811 03:09:53.739551 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.739560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.740132 (+ 572us) server_negotiation.cc:407] Connection header received
0811 03:09:53.740289 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.740293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.740399 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.740556 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:09:53.741111 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.741906 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.743721 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.744252 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.744372 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.745018 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:09:53.745595 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:09:53.745750 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.746087 (+ 337us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.746133 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260811 03:09:53.756217 16860 raft_consensus.cc:1261] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 1 FOLLOWER]: Refusing update from remote peer f66225729d9140ed9d3dc7e927b224fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:53.756217 17000 raft_consensus.cc:1261] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 1 FOLLOWER]: Refusing update from remote peer f66225729d9140ed9d3dc7e927b224fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:53.757436 17219 consensus_queue.cc:1035] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2acd637606a4c5ba2272d75564f3683" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:53.757989 17216 consensus_queue.cc:1035] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1e12dfc3f4345cb8f4006bdd2083ea6" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 39513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:53.770957 17223 mvcc.cc:205] Tried to move back new op lower bound from 7317167283213881344 to 7317167282979450880. Current Snapshot: MvccSnapshot[applied={T|T < 7317167283213881344}]
I20260811 03:09:53.789456 16707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:53.789808 16707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:53.793756 16707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0b3734d33a943ff98440df577cd8ff5: Sending DeleteTablet for 3 replicas of tablet d65aeb1896a44d8084dc21df8141e134
I20260811 03:09:53.795230 17118 tablet_service.cc:1514] Processing DeleteTablet for tablet d65aeb1896a44d8084dc21df8141e134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:53 UTC) from {username='slave'} at 127.0.0.1:50402
I20260811 03:09:53.795578 16836 tablet_service.cc:1514] Processing DeleteTablet for tablet d65aeb1896a44d8084dc21df8141e134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:53 UTC) from {username='slave'} at 127.0.0.1:47120
I20260811 03:09:53.796128 16979 tablet_service.cc:1514] Processing DeleteTablet for tablet d65aeb1896a44d8084dc21df8141e134 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:53 UTC) from {username='slave'} at 127.0.0.1:59674
I20260811 03:09:53.796277 17235 tablet_replica.cc:330] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: stopping tablet replica
I20260811 03:09:53.796927 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 16769
I20260811 03:09:53.801291 17235 raft_consensus.cc:2229] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:09:53.802335 17235 raft_consensus.cc:2258] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:53.805792 17235 ts_tablet_manager.cc:1905] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:53.806648 17237 tablet_replica.cc:330] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: stopping tablet replica
I20260811 03:09:53.807379 17237 raft_consensus.cc:2229] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:53.808108 17237 raft_consensus.cc:2258] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:53.809078 17236 tablet_replica.cc:330] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: stopping tablet replica
I20260811 03:09:53.809588 17236 raft_consensus.cc:2229] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:53.810177 17236 raft_consensus.cc:2258] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:53.811126 17237 ts_tablet_manager.cc:1905] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:53.812675 17236 ts_tablet_manager.cc:1905] T d65aeb1896a44d8084dc21df8141e134 P d2acd637606a4c5ba2272d75564f3683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:53.816641 17235 ts_tablet_manager.cc:1918] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:53.816915 17235 log.cc:1196] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/d65aeb1896a44d8084dc21df8141e134
I20260811 03:09:53.817633 17235 ts_tablet_manager.cc:1939] T d65aeb1896a44d8084dc21df8141e134 P f66225729d9140ed9d3dc7e927b224fb: Deleting consensus metadata
I20260811 03:09:53.819895 16690 catalog_manager.cc:4841] TS f66225729d9140ed9d3dc7e927b224fb (127.27.109.3:39133): tablet d65aeb1896a44d8084dc21df8141e134 (table test-table [id=d4e4dd692e334572b2d330013cf26fe0]) successfully deleted
I20260811 03:09:53.820772 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.798138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.27.109.1:37097)
0811 03:09:53.798338 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.798344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.811669 (+ 13325us) server_negotiation.cc:407] Connection header received
0811 03:09:53.811977 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.811982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.812101 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.812195 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:53.812955 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.814130 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.816284 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.816958 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.817099 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.818853 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:53.818866 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:53.818885 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:53.819718 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:53.819794 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:53.819822 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:53.819914 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.820216 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.820364 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 03:09:53.822154 16901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:53.826797 17237 ts_tablet_manager.cc:1918] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:53.827011 17237 log.cc:1196] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/d65aeb1896a44d8084dc21df8141e134
I20260811 03:09:53.827661 17237 ts_tablet_manager.cc:1939] T d65aeb1896a44d8084dc21df8141e134 P f1e12dfc3f4345cb8f4006bdd2083ea6: Deleting consensus metadata
I20260811 03:09:53.829996 16689 catalog_manager.cc:4841] TS f1e12dfc3f4345cb8f4006bdd2083ea6 (127.27.109.2:39513): tablet d65aeb1896a44d8084dc21df8141e134 (table test-table [id=d4e4dd692e334572b2d330013cf26fe0]) successfully deleted
I20260811 03:09:53.919323 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 16769
W20260811 03:09:53.947244 16689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:37097 (error 108)
I20260811 03:09:53.947326 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 16910
I20260811 03:09:53.959666 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.949143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:47132)
0811 03:09:53.959191 (+ 10048us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:53.959404 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9885,"threads_started":1}
W20260811 03:09:53.960353 16689 catalog_manager.cc:4584] TS d2acd637606a4c5ba2272d75564f3683 (127.27.109.1:37097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d65aeb1896a44d8084dc21df8141e134: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
I20260811 03:09:53.961109 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:53.948443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.27.109.2:39513)
0811 03:09:53.948846 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:53.948852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:53.949485 (+ 633us) server_negotiation.cc:407] Connection header received
0811 03:09:53.949589 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:53.949598 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:53.949744 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:53.949880 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:53.950974 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.952517 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.955547 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:53.956533 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:53.956711 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:53.958737 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:53.958756 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:53.958784 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:53.959688 (+ 904us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:53.959758 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:53.959789 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:53.959912 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:53.960617 (+ 705us) server_negotiation.cc:299] Negotiation successful
0811 03:09:53.960746 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260811 03:09:53.961915 17042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:54.014717 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:54.013773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:47136)
0811 03:09:54.014079 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:54.014210 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260811 03:09:54.016419 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:54.015852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:47140)
0811 03:09:54.016121 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:54.016240 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260811 03:09:54.067323 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 16910
I20260811 03:09:54.078048 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:54.077338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:47154)
0811 03:09:54.077657 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:54.077805 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260811 03:09:54.079727 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:54.079145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:47170)
0811 03:09:54.079362 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:54.079494 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260811 03:09:54.092818 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 17051
I20260811 03:09:54.101251 17206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:54.093947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.27.109.3:39133)
0811 03:09:54.094256 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:54.094273 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:54.094288 (+ 15us) server_negotiation.cc:407] Connection header received
0811 03:09:54.094347 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:54.094351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:54.094444 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:54.094567 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:54.095043 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:54.095845 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:54.097224 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:54.097690 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:54.097843 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:54.099658 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:54.099672 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:54.099697 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:54.100355 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:54.100413 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:54.100446 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:54.100584 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:54.100752 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 03:09:54.100878 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260811 03:09:54.102125 17184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:54.161407 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:54.160648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:47186)
0811 03:09:54.161081 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:54.161212 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260811 03:09:54.162834 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:54.162247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:47194)
0811 03:09:54.162496 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:54.162589 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260811 03:09:54.235198 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 17051
I20260811 03:09:54.260232 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 16670
I20260811 03:09:54.261562 16737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:54.380465 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:54.379758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37097 (local address 127.0.0.1:60974)
0811 03:09:54.379982 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:54.380093 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 03:09:54.380990 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 16670
2026-08-11T03:09:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3483 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:09:54 dist-test-slave-8tdl krb5kdc[17249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:09:54 dist-test-slave-8tdl krb5kdc[17249](info): set up 2 sockets
Aug 11 03:09:54 dist-test-slave-8tdl krb5kdc[17249](info): commencing operation
krb5kdc: starting...
W20260811 03:09:55.620718 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:09:55 dist-test-slave-8tdl krb5kdc[17249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:09:55.987397 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36261
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:56.359139 17263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:56.359527 17263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:56.359695 17263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:56.367340 17263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:09:56.367519 17263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:56.367637 17263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:56.367748 17263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:09:56.367839 17263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:09:56.378324 17263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:56.379843 17263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:56.381336 17263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:56.394112 17270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:56.394111 17271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:56.394379 17273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:56.395291 17263 server_base.cc:1034] running on GCE node
I20260811 03:09:56.396207 17263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:56.397778 17263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:56.399027 17263 hybrid_clock.cc:648] HybridClock initialized: now 1786417796398991 us; error 51 us; skew 500 ppm
I20260811 03:09:56.399408 17263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:56 dist-test-slave-8tdl krb5kdc[17249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417796, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:56.408391 17263 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:09:56.410296 17263 webserver.cc:460] Webserver started at http://127.27.109.62:45733/ using document root <none> and password file <none>
I20260811 03:09:56.411172 17263 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:56.411293 17263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:56.411574 17263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:56.415225 17263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "473fb1aaff0d4837ba06650531162a4d"
format_stamp: "Formatted at 2026-08-11 03:09:56 on dist-test-slave-8tdl"
server_key: "523403dff85825b62c976526b84154b4"
server_key_iv: "e192ff7148c046d2f3f32b35d97c857d"
server_key_version: "encryptionkey@0"
I20260811 03:09:56.416060 17263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "473fb1aaff0d4837ba06650531162a4d"
format_stamp: "Formatted at 2026-08-11 03:09:56 on dist-test-slave-8tdl"
server_key: "523403dff85825b62c976526b84154b4"
server_key_iv: "e192ff7148c046d2f3f32b35d97c857d"
server_key_version: "encryptionkey@0"
I20260811 03:09:56.422076 17263 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260811 03:09:56.426142 17281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:56.427771 17263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:56.427956 17263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "473fb1aaff0d4837ba06650531162a4d"
format_stamp: "Formatted at 2026-08-11 03:09:56 on dist-test-slave-8tdl"
server_key: "523403dff85825b62c976526b84154b4"
server_key_iv: "e192ff7148c046d2f3f32b35d97c857d"
server_key_version: "encryptionkey@0"
I20260811 03:09:56.428143 17263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:56.452386 17263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:56.457152 17263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:56.457419 17263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:56.477411 17263 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:36261
I20260811 03:09:56.477447 17335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:36261 every 8 connection(s)
I20260811 03:09:56.479013 17263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:09:56.483364 17336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:56.485347 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 17263
I20260811 03:09:56.485574 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:09:56.486217 28084 external_mini_cluster.cc:1428] Setting key 781e29f5d2720f9c06bd4f0c926b7e9e
I20260811 03:09:56.500519 17336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d: Bootstrap starting.
Aug 11 03:09:56 dist-test-slave-8tdl krb5kdc[17249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:56.504772 17336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:56.506378 17336 log.cc:824] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:56.509860 17336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d: No bootstrap required, opened a new log
I20260811 03:09:56.512547 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:56.488367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.27.109.62:36261)
0811 03:09:56.489305 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:56.489318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:56.489367 (+ 49us) server_negotiation.cc:407] Connection header received
0811 03:09:56.491140 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:56.491165 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:56.491607 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:56.491970 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:56.494121 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:56.495565 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:56.497671 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:56.498395 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:56.498552 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:56.504192 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:56.504201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:56.504219 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:56.504264 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:56.507956 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:56.508827 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:56.508843 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:56.508850 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:56.509039 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:56.509474 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:56.509479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:56.509484 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:56.510132 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:56.510483 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:56.511005 (+ 522us) server_negotiation.cc:299] Negotiation successful
0811 03:09:56.511471 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":436,"threads_started":1}
I20260811 03:09:56.519127 17336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fb1aaff0d4837ba06650531162a4d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36261 } }
I20260811 03:09:56.519609 17336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:56.519737 17336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473fb1aaff0d4837ba06650531162a4d, State: Initialized, Role: FOLLOWER
I20260811 03:09:56.520546 17336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fb1aaff0d4837ba06650531162a4d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36261 } }
I20260811 03:09:56.520864 17336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:09:56.521039 17336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:09:56.521281 17336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:56.523933 17336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fb1aaff0d4837ba06650531162a4d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36261 } }
I20260811 03:09:56.524513 17336 leader_election.cc:302] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 473fb1aaff0d4837ba06650531162a4d; no voters:
I20260811 03:09:56.525239 17336 leader_election.cc:288] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:09:56.525468 17341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:56.526319 17341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [term 1 LEADER]: Becoming Leader. State: Replica: 473fb1aaff0d4837ba06650531162a4d, State: Running, Role: LEADER
I20260811 03:09:56.527006 17341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fb1aaff0d4837ba06650531162a4d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36261 } }
I20260811 03:09:56.528139 17336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:09:56.530174 17342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "473fb1aaff0d4837ba06650531162a4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fb1aaff0d4837ba06650531162a4d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36261 } } }
I20260811 03:09:56.530136 17344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 473fb1aaff0d4837ba06650531162a4d. Latest consensus state: current_term: 1 leader_uuid: "473fb1aaff0d4837ba06650531162a4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473fb1aaff0d4837ba06650531162a4d" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36261 } } }
I20260811 03:09:56.530813 17344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:56.530813 17342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:56.531785 17347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:09:56.540952 17347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:09:56.551788 17347 catalog_manager.cc:1347] Generated new cluster ID: 0198d0b9e21a461dbbb90ff120c9c0fc
I20260811 03:09:56.552009 17347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:09:56.565721 17347 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:09:56.567332 17347 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:09:56.579608 17347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d: Generated new TSK 0
I20260811 03:09:56.580598 17347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.109.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
Entry for principal kudu/127.27.109.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab.
I20260811 03:09:56.662999 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36261
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:57.030993 17368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:57.031445 17368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:57.031639 17368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:57.039718 17368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:57.039839 17368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:57.040024 17368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:09:57.050540 17368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.1.keytab
--principal=kudu/127.27.109.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:57.052078 17368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:57.053462 17368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:57.067286 17373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:57.067353 17374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:57.067454 17376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:57.068763 17368 server_base.cc:1034] running on GCE node
I20260811 03:09:57.069561 17368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:57.070657 17368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:57.072021 17368 hybrid_clock.cc:648] HybridClock initialized: now 1786417797071939 us; error 54 us; skew 500 ppm
I20260811 03:09:57.072422 17368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:57 dist-test-slave-8tdl krb5kdc[17249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:57.076320 17368 init.cc:377] Logged in from keytab as kudu/127.27.109.1@KRBTEST.COM (short username kudu)
I20260811 03:09:57.077960 17368 webserver.cc:460] Webserver started at http://127.27.109.1:38111/ using document root <none> and password file <none>
I20260811 03:09:57.078871 17368 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:57.079046 17368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:57.079401 17368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:57.082782 17368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "1f770eda8a0b42a48f66487e33055a3c"
format_stamp: "Formatted at 2026-08-11 03:09:57 on dist-test-slave-8tdl"
server_key: "38bd63b60c8e0fd34df0a03900d7a3d9"
server_key_iv: "2e7e08f4363eb6ce5cfb901c5037e258"
server_key_version: "encryptionkey@0"
I20260811 03:09:57.083700 17368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "1f770eda8a0b42a48f66487e33055a3c"
format_stamp: "Formatted at 2026-08-11 03:09:57 on dist-test-slave-8tdl"
server_key: "38bd63b60c8e0fd34df0a03900d7a3d9"
server_key_iv: "2e7e08f4363eb6ce5cfb901c5037e258"
server_key_version: "encryptionkey@0"
I20260811 03:09:57.089830 17368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:09:57.093871 17383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:57.095498 17368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:57.095709 17368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "1f770eda8a0b42a48f66487e33055a3c"
format_stamp: "Formatted at 2026-08-11 03:09:57 on dist-test-slave-8tdl"
server_key: "38bd63b60c8e0fd34df0a03900d7a3d9"
server_key_iv: "2e7e08f4363eb6ce5cfb901c5037e258"
server_key_version: "encryptionkey@0"
I20260811 03:09:57.095953 17368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:57.116379 17368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:57.121580 17368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:57.121945 17368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:57.123123 17368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:57.124708 17368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:57.124823 17368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:57.125003 17368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:57.125087 17368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:57.162886 17368 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:46609
I20260811 03:09:57.163069 17508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:46609 every 8 connection(s)
I20260811 03:09:57.164345 17368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:09:57.170287 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 17368
I20260811 03:09:57.170547 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:09:57.171077 28084 external_mini_cluster.cc:1428] Setting key 1297499c26a425f967da8a132afd89f3
Aug 11 03:09:57 dist-test-slave-8tdl krb5kdc[17249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.1@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:57.186367 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:57.167172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:45133 (local address 127.27.109.62:36261)
0811 03:09:57.167678 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:57.167683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:57.168611 (+ 928us) server_negotiation.cc:407] Connection header received
0811 03:09:57.170575 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:57.170581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:57.170709 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:57.170844 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:57.172607 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:57.173559 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:57.175294 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:57.175728 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:57.175870 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:57.179715 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:57.179721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:57.179724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:57.179761 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:57.182538 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:57.183378 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:57.183386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:57.183389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:57.183526 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:57.184008 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:57.184011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:57.184014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:57.184291 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:57.184468 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:57.185490 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0811 03:09:57.185808 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":276,"threads_started":1}
I20260811 03:09:57.187351 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:57.167684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36261 (local address 127.27.109.1:45133)
0811 03:09:57.168447 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:57.168485 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:57.170251 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:57.171104 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:57.171113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:57.171765 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:57.172426 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:57.172446 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:57.173742 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:57.173746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:57.175134 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:57.175148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:57.176003 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:57.176011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:57.176261 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:57.177051 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:57.177082 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:57.179450 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:57.182730 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:57.182740 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:57.182761 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:57.183237 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:57.183650 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:57.183655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:57.183659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:57.183868 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:57.184447 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:57.184455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:57.184974 (+ 519us) client_negotiation.cc:769] Sending connection context
0811 03:09:57.185271 (+ 297us) client_negotiation.cc:240] Negotiation successful
0811 03:09:57.185838 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":343,"threads_started":1}
I20260811 03:09:57.189391 17509 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36261
I20260811 03:09:57.189755 17509 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:57.190613 17509 heartbeater.cc:511] Master 127.27.109.62:36261 requested a full tablet report, sending...
I20260811 03:09:57.193174 17299 ts_manager.cc:194] Registered new tserver with Master: 1f770eda8a0b42a48f66487e33055a3c (127.27.109.1:46609)
I20260811 03:09:57.195525 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.1@KRBTEST.COM'} at 127.27.109.1:45133
WARNING: no policy specified for kudu/127.27.109.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
Entry for principal kudu/127.27.109.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab.
I20260811 03:09:57.251746 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36261
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:09:57.626255 17515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:57.626556 17515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:57.626780 17515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:57.634441 17515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:57.634562 17515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:57.634785 17515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:09:57.645020 17515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.2.keytab
--principal=kudu/127.27.109.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:57.646409 17515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:57.647866 17515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:57.663060 17520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:57.663105 17521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:57.663076 17523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:57.663938 17515 server_base.cc:1034] running on GCE node
I20260811 03:09:57.664573 17515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:57.665568 17515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:57.666822 17515 hybrid_clock.cc:648] HybridClock initialized: now 1786417797666743 us; error 54 us; skew 500 ppm
I20260811 03:09:57.667204 17515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:57 dist-test-slave-8tdl krb5kdc[17249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:57.670892 17515 init.cc:377] Logged in from keytab as kudu/127.27.109.2@KRBTEST.COM (short username kudu)
I20260811 03:09:57.672578 17515 webserver.cc:460] Webserver started at http://127.27.109.2:45897/ using document root <none> and password file <none>
I20260811 03:09:57.673336 17515 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:57.673494 17515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:57.673837 17515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:57.677150 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "23618d51f5dc487c89459cc43091f994"
format_stamp: "Formatted at 2026-08-11 03:09:57 on dist-test-slave-8tdl"
server_key: "6f5209903d47e37bdef71ee01a537a12"
server_key_iv: "2a7e4b3aa8633f9d2f31ff0cb150a11d"
server_key_version: "encryptionkey@0"
I20260811 03:09:57.678025 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "23618d51f5dc487c89459cc43091f994"
format_stamp: "Formatted at 2026-08-11 03:09:57 on dist-test-slave-8tdl"
server_key: "6f5209903d47e37bdef71ee01a537a12"
server_key_iv: "2a7e4b3aa8633f9d2f31ff0cb150a11d"
server_key_version: "encryptionkey@0"
I20260811 03:09:57.683830 17515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260811 03:09:57.688148 17531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:57.689929 17515 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260811 03:09:57.690122 17515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "23618d51f5dc487c89459cc43091f994"
format_stamp: "Formatted at 2026-08-11 03:09:57 on dist-test-slave-8tdl"
server_key: "6f5209903d47e37bdef71ee01a537a12"
server_key_iv: "2a7e4b3aa8633f9d2f31ff0cb150a11d"
server_key_version: "encryptionkey@0"
I20260811 03:09:57.690353 17515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:57.714722 17515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:57.719697 17515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:57.720043 17515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:57.721594 17515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:57.723946 17515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:57.724073 17515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:57.724229 17515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:57.724308 17515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:57.767007 17515 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:44235
I20260811 03:09:57.767117 17655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:44235 every 8 connection(s)
I20260811 03:09:57.768529 17515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:09:57.769042 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 17515
I20260811 03:09:57.769254 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:09:57.769938 28084 external_mini_cluster.cc:1428] Setting key 457823ba176dc951f4dd34ca30795038
Aug 11 03:09:57 dist-test-slave-8tdl krb5kdc[17249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.2@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:57.792693 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:57.771535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:58801 (local address 127.27.109.62:36261)
0811 03:09:57.772095 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:57.772100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:57.773435 (+ 1335us) server_negotiation.cc:407] Connection header received
0811 03:09:57.775665 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:57.775673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:57.775779 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:57.775907 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:57.777623 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:57.778838 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:57.780440 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:57.781214 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:57.781366 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:57.785625 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:57.785631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:57.785635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:57.785676 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:57.788915 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:57.789987 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:57.789991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:57.789994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:57.790146 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:57.790640 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:57.790647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:57.790655 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:57.791111 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:57.791283 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:57.792035 (+ 752us) server_negotiation.cc:299] Negotiation successful
0811 03:09:57.792300 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":277,"threads_started":1}
I20260811 03:09:57.793854 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:57.772269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36261 (local address 127.27.109.2:58801)
0811 03:09:57.773249 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:57.773300 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:57.775365 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:57.776216 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:57.776225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:57.776771 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:57.777416 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:57.777436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:57.778990 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:57.778994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:57.780279 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:57.780293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:57.781441 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:57.781446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:57.781661 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:57.782417 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:57.782443 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:57.785306 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:57.789144 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:57.789156 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:57.789176 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:57.789749 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:57.790279 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:57.790284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:57.790294 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:57.790508 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:57.791256 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:57.791264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:57.791655 (+ 391us) client_negotiation.cc:769] Sending connection context
0811 03:09:57.791948 (+ 293us) client_negotiation.cc:240] Negotiation successful
0811 03:09:57.792564 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":409,"threads_started":1}
I20260811 03:09:57.795500 17656 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36261
I20260811 03:09:57.795921 17656 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:57.796929 17656 heartbeater.cc:511] Master 127.27.109.62:36261 requested a full tablet report, sending...
I20260811 03:09:57.798910 17299 ts_manager.cc:194] Registered new tserver with Master: 23618d51f5dc487c89459cc43091f994 (127.27.109.2:44235)
I20260811 03:09:57.800046 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.2@KRBTEST.COM'} at 127.27.109.2:58801
WARNING: no policy specified for kudu/127.27.109.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
Entry for principal kudu/127.27.109.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab.
I20260811 03:09:57.850879 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36261
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:09:58.199473 17509 heartbeater.cc:503] Master 127.27.109.62:36261 was elected leader, sending a full tablet report...
W20260811 03:09:58.214746 17663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:58.215132 17663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:58.215292 17663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:58.222913 17663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:58.223075 17663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:58.223258 17663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:09:58.233847 17663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.3.keytab
--principal=kudu/127.27.109.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:36261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:58.235487 17663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:58.236948 17663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:58.251859 17671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:58.251959 17669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:58.251902 17668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:58.253369 17663 server_base.cc:1034] running on GCE node
I20260811 03:09:58.254184 17663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:58.255362 17663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:58.256633 17663 hybrid_clock.cc:648] HybridClock initialized: now 1786417798256587 us; error 54 us; skew 500 ppm
I20260811 03:09:58.257030 17663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:09:58 dist-test-slave-8tdl krb5kdc[17249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417798, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:09:58.260632 17663 init.cc:377] Logged in from keytab as kudu/127.27.109.3@KRBTEST.COM (short username kudu)
I20260811 03:09:58.262516 17663 webserver.cc:460] Webserver started at http://127.27.109.3:40961/ using document root <none> and password file <none>
I20260811 03:09:58.263399 17663 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:58.263576 17663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:58.264057 17663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:58.267426 17663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "767663a856634e419fc9998e0e8500b5"
format_stamp: "Formatted at 2026-08-11 03:09:58 on dist-test-slave-8tdl"
server_key: "4505e9dd3cad5ce554bffa45ddac6ca8"
server_key_iv: "4cd45e2949b4732160a2f8e454fd5cdc"
server_key_version: "encryptionkey@0"
I20260811 03:09:58.268251 17663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "767663a856634e419fc9998e0e8500b5"
format_stamp: "Formatted at 2026-08-11 03:09:58 on dist-test-slave-8tdl"
server_key: "4505e9dd3cad5ce554bffa45ddac6ca8"
server_key_iv: "4cd45e2949b4732160a2f8e454fd5cdc"
server_key_version: "encryptionkey@0"
I20260811 03:09:58.274238 17663 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260811 03:09:58.278519 17678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:58.280138 17663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:58.280357 17663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "767663a856634e419fc9998e0e8500b5"
format_stamp: "Formatted at 2026-08-11 03:09:58 on dist-test-slave-8tdl"
server_key: "4505e9dd3cad5ce554bffa45ddac6ca8"
server_key_iv: "4cd45e2949b4732160a2f8e454fd5cdc"
server_key_version: "encryptionkey@0"
I20260811 03:09:58.280586 17663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:58.310271 17663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:58.315719 17663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:58.316098 17663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:58.317320 17663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:09:58.319046 17663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:09:58.319139 17663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:58.319290 17663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:09:58.319348 17663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:58.360684 17663 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:35207
I20260811 03:09:58.360739 17797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:35207 every 8 connection(s)
I20260811 03:09:58.362190 17663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:09:58.367784 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 17663
I20260811 03:09:58.368016 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:09:58.368605 28084 external_mini_cluster.cc:1428] Setting key 6f2fc3f7168776cf7e95d06ff7864682
Aug 11 03:09:58 dist-test-slave-8tdl krb5kdc[17249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417798, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.3@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:09:58.385408 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.365433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:35423 (local address 127.27.109.62:36261)
0811 03:09:58.366058 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.366068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.366741 (+ 673us) server_negotiation.cc:407] Connection header received
0811 03:09:58.369372 (+ 2631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.369378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.369484 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.369608 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:58.371414 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.372426 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.374336 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.374821 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.374992 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.378775 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:58.378780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:58.378783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:58.378820 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:58.381666 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:58.382718 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:58.382727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:58.382733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:58.382882 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:58.383367 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:58.383373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:58.383376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:58.383674 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:58.383845 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.384723 (+ 878us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.384980 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260811 03:09:58.386539 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.365693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36261 (local address 127.27.109.3:35423)
0811 03:09:58.366512 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.366560 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:58.369055 (+ 2495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:58.369861 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:58.369869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:58.370566 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:09:58.371240 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.371258 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.372794 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.372798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.374164 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.374179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.375091 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.375101 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.375375 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.376146 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:09:58.376172 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:09:58.378489 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:09:58.381842 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:58.381857 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:58.381876 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:58.382556 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:58.383018 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:09:58.383025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:09:58.383029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:09:58.383245 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:09:58.383836 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:09:58.383845 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:09:58.384289 (+ 444us) client_negotiation.cc:769] Sending connection context
0811 03:09:58.384619 (+ 330us) client_negotiation.cc:240] Negotiation successful
0811 03:09:58.385223 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":339,"threads_started":1}
I20260811 03:09:58.388087 17799 heartbeater.cc:347] Connected to a master server at 127.27.109.62:36261
I20260811 03:09:58.388454 17799 heartbeater.cc:464] Registering TS with master...
I20260811 03:09:58.389254 17799 heartbeater.cc:511] Master 127.27.109.62:36261 requested a full tablet report, sending...
I20260811 03:09:58.391227 17299 ts_manager.cc:194] Registered new tserver with Master: 767663a856634e419fc9998e0e8500b5 (127.27.109.3:35207)
I20260811 03:09:58.392124 17299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.109.3@KRBTEST.COM'} at 127.27.109.3:35423
I20260811 03:09:58.397482 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:09:58.419402 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.402721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.27.109.62:36261)
0811 03:09:58.403096 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.403105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.404099 (+ 994us) server_negotiation.cc:407] Connection header received
0811 03:09:58.404225 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.404229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.404314 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.404413 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:58.406080 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.407123 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.408720 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.409215 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.409342 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.412448 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:58.412454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:58.412458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:58.412501 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:58.416000 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:58.416951 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:58.416956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:58.416959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:58.417116 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:58.417520 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:58.417524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:58.417526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:58.418026 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:58.418250 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.418761 (+ 511us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.419106 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 03:09:58.424548 17299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:09:58.431891 17299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:09:58.470739 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.446588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.27.109.3:35207)
0811 03:09:58.458251 (+ 11663us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.458265 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.461081 (+ 2816us) server_negotiation.cc:407] Connection header received
0811 03:09:58.461394 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.461402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.461692 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.461888 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:58.462974 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.464895 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.467443 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.468430 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.468619 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.468887 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.470122 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.470320 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11376,"thread_start_us":12096,"threads_started":1}
I20260811 03:09:58.471481 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.446080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:35207 (local address 127.0.0.1:40800)
0811 03:09:58.460957 (+ 14877us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.460975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:58.461195 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:58.462114 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:58.462121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:58.462178 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:58.462741 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.462756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.465058 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.465063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.467257 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.467270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.468678 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.468685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.469326 (+ 641us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.469407 (+ 81us) client_negotiation.cc:769] Sending connection context
0811 03:09:58.469542 (+ 135us) client_negotiation.cc:240] Negotiation successful
0811 03:09:58.469739 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14740,"mutex_wait_us":4,"spinlock_wait_cycles":23168,"thread_start_us":12422,"threads_started":1}
I20260811 03:09:58.471793 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.445909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:44235 (local address 127.0.0.1:38326)
0811 03:09:58.459297 (+ 13388us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.459369 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:58.459614 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:58.462374 (+ 2760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:58.462379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:58.462416 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:58.463039 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.463056 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.465650 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.465656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.468756 (+ 3100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.468773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.470660 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.470667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.470910 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.470944 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 03:09:58.471023 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 03:09:58.471172 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12742,"mutex_wait_us":172,"thread_start_us":15954,"threads_started":1}
I20260811 03:09:58.471859 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.446122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.27.109.2:44235)
0811 03:09:58.461441 (+ 15319us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.461454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.461491 (+ 37us) server_negotiation.cc:407] Connection header received
0811 03:09:58.461608 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.461617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.461893 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.463311 (+ 1418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:58.463623 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.465471 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.468986 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.470432 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.470604 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.470855 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.471170 (+ 315us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.471365 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15106,"thread_start_us":15945,"threads_started":1}
I20260811 03:09:58.472617 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.449273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.27.109.1:46609)
0811 03:09:58.460457 (+ 11184us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.460469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.461837 (+ 1368us) server_negotiation.cc:407] Connection header received
0811 03:09:58.462415 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.462424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.462653 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.462947 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:58.463624 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.465479 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.469016 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.470553 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.470723 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.471100 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.472120 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.472294 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10975,"thread_start_us":10783,"threads_started":1}
I20260811 03:09:58.473281 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.448462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36664)
0811 03:09:58.461698 (+ 13236us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.461718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:58.462198 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:58.462934 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:58.462946 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:58.462980 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:58.463398 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.463416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.465652 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.465657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.468766 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.468777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.471266 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.471274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.471550 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.471696 (+ 146us) client_negotiation.cc:769] Sending connection context
0811 03:09:58.471798 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 03:09:58.472983 (+ 1185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13107,"mutex_wait_us":158,"spinlock_wait_cycles":896,"thread_start_us":12182,"threads_started":1}
I20260811 03:09:58.476316 17728 tablet_service.cc:1467] Processing CreateTablet for tablet ba69fb0df31b46b3b267d1e53ebacc03 (DEFAULT_TABLE table=test-table [id=6022777c8b67427d918d8850794e64db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:58.477305 17436 tablet_service.cc:1467] Processing CreateTablet for tablet ba69fb0df31b46b3b267d1e53ebacc03 (DEFAULT_TABLE table=test-table [id=6022777c8b67427d918d8850794e64db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:58.477305 17582 tablet_service.cc:1467] Processing CreateTablet for tablet ba69fb0df31b46b3b267d1e53ebacc03 (DEFAULT_TABLE table=test-table [id=6022777c8b67427d918d8850794e64db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:09:58.478120 17728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba69fb0df31b46b3b267d1e53ebacc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:58.478765 17436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba69fb0df31b46b3b267d1e53ebacc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:58.478768 17582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba69fb0df31b46b3b267d1e53ebacc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:58.497824 17820 tablet_bootstrap.cc:492] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: Bootstrap starting.
I20260811 03:09:58.505909 17820 tablet_bootstrap.cc:654] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:58.507666 17820 log.cc:824] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:58.509446 17819 tablet_bootstrap.cc:492] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Bootstrap starting.
I20260811 03:09:58.510653 17821 tablet_bootstrap.cc:492] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: Bootstrap starting.
I20260811 03:09:58.513055 17819 tablet_bootstrap.cc:654] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:58.513983 17821 tablet_bootstrap.cc:654] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:58.514274 17819 log.cc:824] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:58.515465 17821 log.cc:824] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:58.517068 17820 tablet_bootstrap.cc:492] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: No bootstrap required, opened a new log
I20260811 03:09:58.517413 17820 ts_tablet_manager.cc:1397] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.003s
I20260811 03:09:58.518325 17819 tablet_bootstrap.cc:492] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: No bootstrap required, opened a new log
I20260811 03:09:58.518626 17819 ts_tablet_manager.cc:1397] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260811 03:09:58.523568 17821 tablet_bootstrap.cc:492] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: No bootstrap required, opened a new log
I20260811 03:09:58.523926 17821 ts_tablet_manager.cc:1397] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260811 03:09:58.525252 17820 raft_consensus.cc:348] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.525689 17820 raft_consensus.cc:374] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:58.525794 17820 raft_consensus.cc:729] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23618d51f5dc487c89459cc43091f994, State: Initialized, Role: FOLLOWER
I20260811 03:09:58.526437 17820 consensus_queue.cc:260] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.527855 17656 heartbeater.cc:503] Master 127.27.109.62:36261 was elected leader, sending a full tablet report...
I20260811 03:09:58.528668 17820 ts_tablet_manager.cc:1428] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: Time spent starting tablet: real 0.011s user 0.012s sys 0.001s
I20260811 03:09:58.530068 17819 raft_consensus.cc:348] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.530570 17819 raft_consensus.cc:374] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:58.530719 17819 raft_consensus.cc:729] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767663a856634e419fc9998e0e8500b5, State: Initialized, Role: FOLLOWER
I20260811 03:09:58.531539 17819 consensus_queue.cc:260] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.533710 17799 heartbeater.cc:503] Master 127.27.109.62:36261 was elected leader, sending a full tablet report...
I20260811 03:09:58.534549 17819 ts_tablet_manager.cc:1428] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Time spent starting tablet: real 0.016s user 0.012s sys 0.002s
I20260811 03:09:58.536525 17821 raft_consensus.cc:348] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.537006 17821 raft_consensus.cc:374] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:58.537147 17821 raft_consensus.cc:729] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f770eda8a0b42a48f66487e33055a3c, State: Initialized, Role: FOLLOWER
I20260811 03:09:58.537810 17821 consensus_queue.cc:260] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.539621 17821 ts_tablet_manager.cc:1428] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260811 03:09:58.572885 17825 raft_consensus.cc:482] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:09:58.573182 17825 raft_consensus.cc:504] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.575019 17825 leader_election.cc:288] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 767663a856634e419fc9998e0e8500b5 (127.27.109.3:35207), 1f770eda8a0b42a48f66487e33055a3c (127.27.109.1:46609)
I20260811 03:09:58.585386 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.575658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:58591 (local address 127.27.109.3:35207)
0811 03:09:58.575910 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.575919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.576586 (+ 667us) server_negotiation.cc:407] Connection header received
0811 03:09:58.576797 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.576803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.576883 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.576988 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:58.577896 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.579107 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.582636 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.583943 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.584131 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.584212 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.584980 (+ 768us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.585120 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 03:09:58.585512 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.575558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:59599 (local address 127.27.109.1:46609)
0811 03:09:58.575804 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.575813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.576589 (+ 776us) server_negotiation.cc:407] Connection header received
0811 03:09:58.576797 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.576803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.576884 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.576988 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:09:58.577891 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.579107 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.582636 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.583963 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.584131 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.584212 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.585066 (+ 854us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.585263 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 03:09:58.585836 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.575320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:35207 (local address 127.27.109.2:58591)
0811 03:09:58.576466 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.576484 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:58.576639 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:58.577056 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:58.577062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:58.577108 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:58.577716 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.577728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.579275 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.579279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.582412 (+ 3133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.582433 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.584298 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.584304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.584660 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.584806 (+ 146us) client_negotiation.cc:769] Sending connection context
0811 03:09:58.584938 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 03:09:58.585120 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":420,"spinlock_wait_cycles":27136,"thread_start_us":665,"threads_started":1}
I20260811 03:09:58.586199 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.575320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.27.109.2:59599)
0811 03:09:58.576410 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.576433 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:09:58.576639 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:09:58.577056 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:09:58.577062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:09:58.577113 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:09:58.577670 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.577690 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.579269 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.579275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.582412 (+ 3137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:09:58.582433 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.584288 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.584297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:09:58.584661 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.584822 (+ 161us) client_negotiation.cc:769] Sending connection context
0811 03:09:58.584981 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 03:09:58.585162 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":357,"spinlock_wait_cycles":48128,"thread_start_us":619,"threads_started":1}
I20260811 03:09:58.587137 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba69fb0df31b46b3b267d1e53ebacc03" candidate_uuid: "23618d51f5dc487c89459cc43091f994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767663a856634e419fc9998e0e8500b5" is_pre_election: true
I20260811 03:09:58.587244 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba69fb0df31b46b3b267d1e53ebacc03" candidate_uuid: "23618d51f5dc487c89459cc43091f994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f770eda8a0b42a48f66487e33055a3c" is_pre_election: true
I20260811 03:09:58.587720 17749 raft_consensus.cc:2454] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23618d51f5dc487c89459cc43091f994 in term 0.
I20260811 03:09:58.587882 17459 raft_consensus.cc:2454] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23618d51f5dc487c89459cc43091f994 in term 0.
I20260811 03:09:58.588434 17534 leader_election.cc:302] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23618d51f5dc487c89459cc43091f994, 767663a856634e419fc9998e0e8500b5; no voters:
I20260811 03:09:58.588915 17825 raft_consensus.cc:2781] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:09:58.589079 17825 raft_consensus.cc:482] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:09:58.589221 17825 raft_consensus.cc:3037] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:58.592183 17825 raft_consensus.cc:504] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.593039 17825 leader_election.cc:288] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [CANDIDATE]: Term 1 election: Requested vote from peers 767663a856634e419fc9998e0e8500b5 (127.27.109.3:35207), 1f770eda8a0b42a48f66487e33055a3c (127.27.109.1:46609)
I20260811 03:09:58.593605 17749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba69fb0df31b46b3b267d1e53ebacc03" candidate_uuid: "23618d51f5dc487c89459cc43091f994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767663a856634e419fc9998e0e8500b5"
I20260811 03:09:58.593674 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba69fb0df31b46b3b267d1e53ebacc03" candidate_uuid: "23618d51f5dc487c89459cc43091f994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f770eda8a0b42a48f66487e33055a3c"
I20260811 03:09:58.593874 17749 raft_consensus.cc:3037] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:58.593948 17459 raft_consensus.cc:3037] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:58.596735 17459 raft_consensus.cc:2454] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23618d51f5dc487c89459cc43091f994 in term 1.
I20260811 03:09:58.596880 17749 raft_consensus.cc:2454] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23618d51f5dc487c89459cc43091f994 in term 1.
I20260811 03:09:58.597313 17532 leader_election.cc:302] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f770eda8a0b42a48f66487e33055a3c, 23618d51f5dc487c89459cc43091f994; no voters:
I20260811 03:09:58.597729 17825 raft_consensus.cc:2781] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:58.598280 17825 raft_consensus.cc:686] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 1 LEADER]: Becoming Leader. State: Replica: 23618d51f5dc487c89459cc43091f994, State: Running, Role: LEADER
I20260811 03:09:58.598932 17825 consensus_queue.cc:237] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } }
I20260811 03:09:58.605335 17299 catalog_manager.cc:5495] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23618d51f5dc487c89459cc43091f994 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "23618d51f5dc487c89459cc43091f994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23618d51f5dc487c89459cc43091f994" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 44235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 } health_report { overall_health: UNKNOWN } } }
W20260811 03:09:58.614197 17800 tablet.cc:2369] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:09:58.635051 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.628587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.27.109.2:44235)
0811 03:09:58.628965 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.628972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.629113 (+ 141us) server_negotiation.cc:407] Connection header received
0811 03:09:58.629394 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.629399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.629496 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.629695 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:09:58.630154 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.630963 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.632565 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.633019 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.633170 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.633758 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:09:58.634301 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:09:58.634486 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.634552 (+ 66us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.634649 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260811 03:09:58.643841 17459 raft_consensus.cc:1261] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 1 FOLLOWER]: Refusing update from remote peer 23618d51f5dc487c89459cc43091f994: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:58.643900 17749 raft_consensus.cc:1261] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 1 FOLLOWER]: Refusing update from remote peer 23618d51f5dc487c89459cc43091f994: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:09:58.644899 17830 consensus_queue.cc:1035] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "767663a856634e419fc9998e0e8500b5" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:58.645344 17825 consensus_queue.cc:1035] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f770eda8a0b42a48f66487e33055a3c" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 46609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:09:58.677369 17298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:58.677690 17298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:09:58.681967 17298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 473fb1aaff0d4837ba06650531162a4d: Sending DeleteTablet for 3 replicas of tablet ba69fb0df31b46b3b267d1e53ebacc03
I20260811 03:09:58.683647 17436 tablet_service.cc:1514] Processing DeleteTablet for tablet ba69fb0df31b46b3b267d1e53ebacc03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:36664
I20260811 03:09:58.683661 17728 tablet_service.cc:1514] Processing DeleteTablet for tablet ba69fb0df31b46b3b267d1e53ebacc03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:40800
I20260811 03:09:58.683799 17582 tablet_service.cc:1514] Processing DeleteTablet for tablet ba69fb0df31b46b3b267d1e53ebacc03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:09:58 UTC) from {username='kudu', principal='kudu/127.27.109.62@KRBTEST.COM'} at 127.0.0.1:38326
I20260811 03:09:58.691092 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 17368
I20260811 03:09:58.696431 17846 tablet_replica.cc:330] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: stopping tablet replica
I20260811 03:09:58.697117 17846 raft_consensus.cc:2229] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:58.697749 17846 raft_consensus.cc:2258] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:58.698616 17847 tablet_replica.cc:330] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: stopping tablet replica
I20260811 03:09:58.699106 17847 raft_consensus.cc:2229] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:09:58.699658 17847 raft_consensus.cc:2258] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:09:58.702389 17847 ts_tablet_manager.cc:1905] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:58.712447 17846 ts_tablet_manager.cc:1905] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:58.713109 17847 ts_tablet_manager.cc:1918] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:58.713335 17847 log.cc:1196] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/ba69fb0df31b46b3b267d1e53ebacc03
I20260811 03:09:58.713649 17845 tablet_replica.cc:330] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: stopping tablet replica
I20260811 03:09:58.714164 17845 raft_consensus.cc:2229] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:09:58.714648 17847 ts_tablet_manager.cc:1939] T ba69fb0df31b46b3b267d1e53ebacc03 P 767663a856634e419fc9998e0e8500b5: Deleting consensus metadata
I20260811 03:09:58.715128 17845 raft_consensus.cc:2258] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 11 03:09:58 dist-test-slave-8tdl krb5kdc[17249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.1@KRBTEST.COM
I20260811 03:09:58.716938 17284 catalog_manager.cc:4841] TS 767663a856634e419fc9998e0e8500b5 (127.27.109.3:35207): tablet ba69fb0df31b46b3b267d1e53ebacc03 (table test-table [id=6022777c8b67427d918d8850794e64db]) successfully deleted
I20260811 03:09:58.720671 17845 ts_tablet_manager.cc:1905] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:09:58.728078 17845 ts_tablet_manager.cc:1918] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:58.728365 17845 log.cc:1196] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/ba69fb0df31b46b3b267d1e53ebacc03
I20260811 03:09:58.729207 17845 ts_tablet_manager.cc:1939] T ba69fb0df31b46b3b267d1e53ebacc03 P 23618d51f5dc487c89459cc43091f994: Deleting consensus metadata
I20260811 03:09:58.731499 17285 catalog_manager.cc:4841] TS 23618d51f5dc487c89459cc43091f994 (127.27.109.2:44235): tablet ba69fb0df31b46b3b267d1e53ebacc03 (table test-table [id=6022777c8b67427d918d8850794e64db]) successfully deleted
I20260811 03:09:58.733594 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.695317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.27.109.1:46609)
0811 03:09:58.695591 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.695600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.703291 (+ 7691us) server_negotiation.cc:407] Connection header received
0811 03:09:58.703550 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.703559 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.703691 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.703822 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:58.706378 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.707954 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.710729 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.711654 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.711791 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.718629 (+ 6838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:58.718652 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:58.718705 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:58.718762 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:58.726403 (+ 7641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:58.728997 (+ 2594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:58.729068 (+ 71us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:58.729072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:58.729332 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:58.730805 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:58.730811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:58.730814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:58.731881 (+ 1067us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:58.732263 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.732728 (+ 465us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.733149 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:09:58.734628 17503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:58.846311 17846 ts_tablet_manager.cc:1918] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:09:58.846644 17846 log.cc:1196] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786417454123933-28084-0/minicluster-data/ts-0/wal/wals/ba69fb0df31b46b3b267d1e53ebacc03
I20260811 03:09:58.847180 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 17368
I20260811 03:09:58.847362 17846 ts_tablet_manager.cc:1939] T ba69fb0df31b46b3b267d1e53ebacc03 P 1f770eda8a0b42a48f66487e33055a3c: Deleting consensus metadata
I20260811 03:09:58.870194 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 17515
W20260811 03:09:58.870203 17282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:46609 (error 108)
I20260811 03:09:58.872547 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.871972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36682)
0811 03:09:58.872184 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.872330 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260811 03:09:58.872915 17282 catalog_manager.cc:4584] TS 1f770eda8a0b42a48f66487e33055a3c (127.27.109.1:46609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba69fb0df31b46b3b267d1e53ebacc03: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Aug 11 03:09:58 dist-test-slave-8tdl krb5kdc[17249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.2@KRBTEST.COM
I20260811 03:09:58.889247 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.871572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.27.109.2:44235)
0811 03:09:58.871806 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:58.871812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:58.871837 (+ 25us) server_negotiation.cc:407] Connection header received
0811 03:09:58.872383 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:58.872387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:58.872469 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:58.872586 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:58.874161 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.875105 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.876423 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:58.876881 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:58.877033 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:58.881578 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:58.881595 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:58.881611 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:58.881654 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:58.886001 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:58.886872 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:58.886881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:58.886883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:58.887051 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:58.887479 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:58.887484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:58.887487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:58.887885 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:58.888288 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:58.888547 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 03:09:58.888818 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 03:09:58.889979 17650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:58.926939 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.926223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36696)
0811 03:09:58.926506 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.926709 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260811 03:09:58.928224 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.927761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36706)
0811 03:09:58.927947 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.928061 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260811 03:09:58.989182 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.988597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36712)
0811 03:09:58.988852 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.989013 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260811 03:09:58.990486 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:58.990078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36722)
0811 03:09:58.990259 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:58.990338 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260811 03:09:59.024048 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 17515
I20260811 03:09:59.048086 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 17663
Aug 11 03:09:59 dist-test-slave-8tdl krb5kdc[17249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.3@KRBTEST.COM
I20260811 03:09:59.065302 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:59.049009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.27.109.3:35207)
0811 03:09:59.049229 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:59.049234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:59.049256 (+ 22us) server_negotiation.cc:407] Connection header received
0811 03:09:59.049319 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:59.049323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:59.049406 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:59.049509 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:59.050941 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:59.051687 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:59.053096 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:59.053640 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:59.053793 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:59.059291 (+ 5498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:59.059313 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:59.059338 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:09:59.059426 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:59.062775 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:59.063487 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:59.063495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:59.063497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:59.063651 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:09:59.063991 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:09:59.063994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:09:59.063996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:09:59.064356 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:59.064602 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:59.064841 (+ 239us) server_negotiation.cc:299] Negotiation successful
0811 03:09:59.065031 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 03:09:59.066048 17793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:59.071086 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:59.070581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36734)
0811 03:09:59.070784 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:59.070881 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 03:09:59.072245 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:59.071864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36746)
0811 03:09:59.072058 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:59.072138 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260811 03:09:59.160827 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 17663
I20260811 03:09:59.183573 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 17263
I20260811 03:09:59.184825 17331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:09:59.301044 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:59.300446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:46609 (local address 127.0.0.1:36762)
0811 03:09:59.300694 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0811 03:09:59.300802 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:46609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260811 03:09:59.301481 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 17263
2026-08-11T03:09:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4924 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-11T03:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:09:59Z Disabled control of system clock
I20260811 03:09:59.410387 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34805
--webserver_interface=127.27.109.62
--webserver_port=0
--builtin_ntp_servers=127.27.109.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 03:09:59.783213 17858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:09:59.783581 17858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:09:59.783768 17858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:09:59.791606 17858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:09:59.791752 17858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:09:59.791862 17858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:09:59.791985 17858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:09:59.792071 17858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:09:59.802379 17858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:09:59.803958 17858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:09:59.805922 17858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:09:59.819826 17864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:59.819917 17865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:09:59.820323 17867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:09:59.821520 17858 server_base.cc:1034] running on GCE node
I20260811 03:09:59.822553 17858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:09:59.824287 17858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:09:59.825582 17858 hybrid_clock.cc:648] HybridClock initialized: now 1786417799825508 us; error 60 us; skew 500 ppm
I20260811 03:09:59.826102 17858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:09:59.828877 17858 webserver.cc:460] Webserver started at http://127.27.109.62:36021/ using document root <none> and password file <none>
I20260811 03:09:59.829684 17858 fs_manager.cc:359] Metadata directory not provided
I20260811 03:09:59.829833 17858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:09:59.830199 17858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:09:59.833559 17858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "2a640960e0fe45a4be16b54efd739f3a"
format_stamp: "Formatted at 2026-08-11 03:09:59 on dist-test-slave-8tdl"
server_key: "d31b884dcd046f973379681393db3ef5"
server_key_iv: "7b18313d116c1959d4fc4a40934b67de"
server_key_version: "encryptionkey@0"
I20260811 03:09:59.834487 17858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "2a640960e0fe45a4be16b54efd739f3a"
format_stamp: "Formatted at 2026-08-11 03:09:59 on dist-test-slave-8tdl"
server_key: "d31b884dcd046f973379681393db3ef5"
server_key_iv: "7b18313d116c1959d4fc4a40934b67de"
server_key_version: "encryptionkey@0"
I20260811 03:09:59.840669 17858 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260811 03:09:59.845018 17873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:09:59.846752 17858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:09:59.847020 17858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "2a640960e0fe45a4be16b54efd739f3a"
format_stamp: "Formatted at 2026-08-11 03:09:59 on dist-test-slave-8tdl"
server_key: "d31b884dcd046f973379681393db3ef5"
server_key_iv: "7b18313d116c1959d4fc4a40934b67de"
server_key_version: "encryptionkey@0"
I20260811 03:09:59.847330 17858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:09:59.870245 17858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:09:59.871291 17858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:09:59.871635 17858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:09:59.893487 17858 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:34805
I20260811 03:09:59.893510 17932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:34805 every 8 connection(s)
I20260811 03:09:59.895256 17858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:09:59.896716 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 17858
I20260811 03:09:59.896906 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:09:59.897574 28084 external_mini_cluster.cc:1428] Setting key f931a267e72e45bd19534239b9f114df
I20260811 03:09:59.900511 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:09:59.913321 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:09:59.899772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.27.109.62:34805)
0811 03:09:59.901170 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0811 03:09:59.901183 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:09:59.901232 (+ 49us) server_negotiation.cc:407] Connection header received
0811 03:09:59.903231 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:09:59.903254 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:09:59.903753 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:09:59.904230 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:09:59.904799 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:59.906245 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:59.908241 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:09:59.908995 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:09:59.909141 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:09:59.910667 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:09:59.910734 (+ 67us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:09:59.910759 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:09:59.911416 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:09:59.911500 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:09:59.911534 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:09:59.911633 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 03:09:59.911896 (+ 263us) server_negotiation.cc:299] Negotiation successful
0811 03:09:59.912300 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":471,"threads_started":1}
I20260811 03:09:59.914484 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a: Bootstrap starting.
I20260811 03:09:59.918111 17933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a: Neither blocks nor log segments found. Creating new log.
I20260811 03:09:59.919327 17933 log.cc:824] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a: Log is configured to *not* fsync() on all Append() calls
I20260811 03:09:59.922953 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a: No bootstrap required, opened a new log
I20260811 03:09:59.931365 17933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a640960e0fe45a4be16b54efd739f3a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34805 } }
I20260811 03:09:59.931865 17933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:09:59.931980 17933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a640960e0fe45a4be16b54efd739f3a, State: Initialized, Role: FOLLOWER
I20260811 03:09:59.932699 17933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a640960e0fe45a4be16b54efd739f3a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34805 } }
I20260811 03:09:59.932945 17933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:09:59.933122 17933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:09:59.933331 17933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:09:59.935772 17933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a640960e0fe45a4be16b54efd739f3a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34805 } }
I20260811 03:09:59.936352 17933 leader_election.cc:302] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a640960e0fe45a4be16b54efd739f3a; no voters:
I20260811 03:09:59.937028 17933 leader_election.cc:288] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:09:59.937228 17938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:09:59.938062 17938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [term 1 LEADER]: Becoming Leader. State: Replica: 2a640960e0fe45a4be16b54efd739f3a, State: Running, Role: LEADER
I20260811 03:09:59.938622 17938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a640960e0fe45a4be16b54efd739f3a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34805 } }
I20260811 03:09:59.939407 17933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:09:59.941766 17939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a640960e0fe45a4be16b54efd739f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a640960e0fe45a4be16b54efd739f3a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34805 } } }
I20260811 03:09:59.942224 17939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:59.942785 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a640960e0fe45a4be16b54efd739f3a. Latest consensus state: current_term: 1 leader_uuid: "2a640960e0fe45a4be16b54efd739f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a640960e0fe45a4be16b54efd739f3a" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34805 } } }
I20260811 03:09:59.943281 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a [sys.catalog]: This master's current role is: LEADER
I20260811 03:09:59.943912 17945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:09:59.957103 17945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:09:59.967473 17945 catalog_manager.cc:1347] Generated new cluster ID: d61dfbfbcec0476788f8cbc29a64f9b2
I20260811 03:09:59.967631 17945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:09:59.987795 17945 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:09:59.989933 17945 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:00.021297 17945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a: Generated new TSK 0
I20260811 03:10:00.022382 17945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:00.029342 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.1:0
--local_ip_for_outbound_sockets=127.27.109.1
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34805
--builtin_ntp_servers=127.27.109.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 03:10:00.406757 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:00.407127 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:00.407286 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:00.415060 17958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:00.415174 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:00.415341 17958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.1
I20260811 03:10:00.425136 17958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.109.1
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.1
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:00.426544 17958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:00.428416 17958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:00.442924 17963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:00.442826 17964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:00.443444 17958 server_base.cc:1034] running on GCE node
W20260811 03:10:00.443544 17966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:00.444409 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:00.445374 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:00.446602 17958 hybrid_clock.cc:648] HybridClock initialized: now 1786417800446563 us; error 34 us; skew 500 ppm
I20260811 03:10:00.446973 17958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:10:00.449497 17958 webserver.cc:460] Webserver started at http://127.27.109.1:35421/ using document root <none> and password file <none>
I20260811 03:10:00.450232 17958 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:00.450325 17958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:00.450600 17958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:00.454150 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/data/instance:
uuid: "0be1179856ed4208bc8be1baa4f0adc5"
format_stamp: "Formatted at 2026-08-11 03:10:00 on dist-test-slave-8tdl"
server_key: "7410da524745e2b223bf2be1e9e2b457"
server_key_iv: "71e499802c4882c2e7f45209c3e5f171"
server_key_version: "encryptionkey@0"
I20260811 03:10:00.455029 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance:
uuid: "0be1179856ed4208bc8be1baa4f0adc5"
format_stamp: "Formatted at 2026-08-11 03:10:00 on dist-test-slave-8tdl"
server_key: "7410da524745e2b223bf2be1e9e2b457"
server_key_iv: "71e499802c4882c2e7f45209c3e5f171"
server_key_version: "encryptionkey@0"
I20260811 03:10:00.460721 17958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:10:00.464558 17973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:00.466114 17958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 03:10:00.466272 17958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/wal
uuid: "0be1179856ed4208bc8be1baa4f0adc5"
format_stamp: "Formatted at 2026-08-11 03:10:00 on dist-test-slave-8tdl"
server_key: "7410da524745e2b223bf2be1e9e2b457"
server_key_iv: "71e499802c4882c2e7f45209c3e5f171"
server_key_version: "encryptionkey@0"
I20260811 03:10:00.466470 17958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:00.481454 17958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:00.482331 17958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:00.482613 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:00.483737 17958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:10:00.485307 17958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:10:00.485399 17958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:00.485479 17958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:10:00.485539 17958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:00.521776 17958 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.1:37805
I20260811 03:10:00.521798 18090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.1:37805 every 8 connection(s)
I20260811 03:10:00.523272 17958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/data/info.pb
I20260811 03:10:00.531304 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 17958
I20260811 03:10:00.531522 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-0/wal/instance
I20260811 03:10:00.531965 28084 external_mini_cluster.cc:1428] Setting key 5e3af0786d6fc898099501cbc3c89e7d
I20260811 03:10:00.535218 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.2:0
--local_ip_for_outbound_sockets=127.27.109.2
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34805
--builtin_ntp_servers=127.27.109.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 03:10:00.537050 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:00.525993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.1:46885 (local address 127.27.109.62:34805)
0811 03:10:00.526481 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:00.526486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:00.527660 (+ 1174us) server_negotiation.cc:407] Connection header received
0811 03:10:00.529553 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:00.529558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:00.529671 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:00.529798 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:00.531048 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:00.531936 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:00.533769 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:00.534256 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:00.534361 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:00.536207 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:00.536213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:00.536216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:00.536236 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:00.536279 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:00.536292 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:00.536372 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:00.536678 (+ 306us) server_negotiation.cc:299] Negotiation successful
0811 03:10:00.536770 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
I20260811 03:10:00.538648 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:00.526515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34805 (local address 127.27.109.1:46885)
0811 03:10:00.527483 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:00.527529 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:00.529258 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:00.530052 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:00.530060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:00.530154 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:00.530854 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:00.530876 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:00.532119 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:00.532123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:00.533603 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:00.533622 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:00.534506 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:00.534514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:00.534883 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:00.535780 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:10:00.535786 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:00.535848 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:10:00.535965 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:00.536400 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:00.536416 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:00.536427 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 03:10:00.536629 (+ 202us) client_negotiation.cc:240] Negotiation successful
0811 03:10:00.537138 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":340,"threads_started":1}
I20260811 03:10:00.540584 18092 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34805
I20260811 03:10:00.540939 18092 heartbeater.cc:464] Registering TS with master...
I20260811 03:10:00.541771 18092 heartbeater.cc:511] Master 127.27.109.62:34805 requested a full tablet report, sending...
I20260811 03:10:00.544188 17892 ts_manager.cc:194] Registered new tserver with Master: 0be1179856ed4208bc8be1baa4f0adc5 (127.27.109.1:37805)
I20260811 03:10:00.545742 17892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.1:46885
W20260811 03:10:00.910056 18096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:00.910445 18096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:00.910625 18096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:00.918620 18096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:00.918836 18096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:00.918972 18096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.2
I20260811 03:10:00.928947 18096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.109.2
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.2
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:00.930418 18096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:00.932703 18096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:00.947765 18102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:00.947836 18103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:00.947861 18105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:00.949170 18096 server_base.cc:1034] running on GCE node
I20260811 03:10:00.949992 18096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:00.951116 18096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:00.952399 18096 hybrid_clock.cc:648] HybridClock initialized: now 1786417800952336 us; error 38 us; skew 500 ppm
I20260811 03:10:00.952860 18096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:10:00.956029 18096 webserver.cc:460] Webserver started at http://127.27.109.2:41251/ using document root <none> and password file <none>
I20260811 03:10:00.956805 18096 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:00.956909 18096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:00.957226 18096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:00.960817 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/data/instance:
uuid: "987937708c1548a79b5ca1411b8048f7"
format_stamp: "Formatted at 2026-08-11 03:10:00 on dist-test-slave-8tdl"
server_key: "7e5bdf6019a80ebb82279748bb2cb90e"
server_key_iv: "ea58cdf4ddf11212b7664d7114dfe95c"
server_key_version: "encryptionkey@0"
I20260811 03:10:00.961746 18096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance:
uuid: "987937708c1548a79b5ca1411b8048f7"
format_stamp: "Formatted at 2026-08-11 03:10:00 on dist-test-slave-8tdl"
server_key: "7e5bdf6019a80ebb82279748bb2cb90e"
server_key_iv: "ea58cdf4ddf11212b7664d7114dfe95c"
server_key_version: "encryptionkey@0"
I20260811 03:10:00.968241 18096 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260811 03:10:00.972401 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:00.974028 18096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:10:00.974185 18096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/wal
uuid: "987937708c1548a79b5ca1411b8048f7"
format_stamp: "Formatted at 2026-08-11 03:10:00 on dist-test-slave-8tdl"
server_key: "7e5bdf6019a80ebb82279748bb2cb90e"
server_key_iv: "ea58cdf4ddf11212b7664d7114dfe95c"
server_key_version: "encryptionkey@0"
I20260811 03:10:00.974388 18096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:00.990187 18096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:00.991178 18096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:00.991480 18096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:00.992563 18096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:10:00.994359 18096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:10:00.994462 18096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:00.994555 18096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:10:00.994611 18096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:01.032883 18096 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.2:36145
I20260811 03:10:01.032939 18233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.2:36145 every 8 connection(s)
I20260811 03:10:01.034348 18096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/data/info.pb
I20260811 03:10:01.044358 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 18096
I20260811 03:10:01.044575 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/wal/instance
I20260811 03:10:01.045189 28084 external_mini_cluster.cc:1428] Setting key 5471f54a33822491a80dbd6291069324
I20260811 03:10:01.048607 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.109.3:0
--local_ip_for_outbound_sockets=127.27.109.3
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34805
--builtin_ntp_servers=127.27.109.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 03:10:01.049161 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.037038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:43867 (local address 127.27.109.62:34805)
0811 03:10:01.037309 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:01.037314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:01.038755 (+ 1441us) server_negotiation.cc:407] Connection header received
0811 03:10:01.041016 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:01.041021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:01.041103 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:01.041204 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:01.042517 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.043392 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.045929 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.046503 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.046659 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.048290 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:01.048295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:01.048299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:01.048319 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:01.048352 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:01.048364 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:01.048465 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:01.048707 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 03:10:01.048818 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260811 03:10:01.050386 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.037607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34805 (local address 127.27.109.2:43867)
0811 03:10:01.038554 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:01.038591 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:01.040721 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:01.041512 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:01.041521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:01.041653 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:01.042312 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.042332 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.043574 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.043578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.045737 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.045755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.046791 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.046800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.047048 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.047923 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:10:01.047926 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:01.047972 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:10:01.048068 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:01.048464 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:01.048478 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:01.048489 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 03:10:01.048695 (+ 206us) client_negotiation.cc:240] Negotiation successful
0811 03:10:01.049142 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":279,"threads_started":1}
I20260811 03:10:01.052026 18234 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34805
I20260811 03:10:01.052436 18234 heartbeater.cc:464] Registering TS with master...
I20260811 03:10:01.053529 18234 heartbeater.cc:511] Master 127.27.109.62:34805 requested a full tablet report, sending...
I20260811 03:10:01.055402 17892 ts_manager.cc:194] Registered new tserver with Master: 987937708c1548a79b5ca1411b8048f7 (127.27.109.2:36145)
I20260811 03:10:01.056694 17892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.2:43867
W20260811 03:10:01.419785 18237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:01.420090 18237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:01.420177 18237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:01.427636 18237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:01.427754 18237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:01.427865 18237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.109.3
I20260811 03:10:01.437453 18237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.109.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.109.3
--webserver_port=0
--tserver_master_addrs=127.27.109.62:34805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.109.3
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:01.439040 18237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:01.441164 18237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:01.456262 18242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:01.456415 18245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:01.456441 18243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:01.457495 18237 server_base.cc:1034] running on GCE node
I20260811 03:10:01.458273 18237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:01.459425 18237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:01.460641 18237 hybrid_clock.cc:648] HybridClock initialized: now 1786417801460569 us; error 59 us; skew 500 ppm
I20260811 03:10:01.460997 18237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 03:10:01.466627 18237 webserver.cc:460] Webserver started at http://127.27.109.3:38087/ using document root <none> and password file <none>
I20260811 03:10:01.467549 18237 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:01.467701 18237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:01.468050 18237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:01.471488 18237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/data/instance:
uuid: "8437209937004cbea2fb31614d745052"
format_stamp: "Formatted at 2026-08-11 03:10:01 on dist-test-slave-8tdl"
server_key: "93da6f932e2d925221834b51923f48a7"
server_key_iv: "16c9b21550f875c96184bd0392670002"
server_key_version: "encryptionkey@0"
I20260811 03:10:01.472357 18237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance:
uuid: "8437209937004cbea2fb31614d745052"
format_stamp: "Formatted at 2026-08-11 03:10:01 on dist-test-slave-8tdl"
server_key: "93da6f932e2d925221834b51923f48a7"
server_key_iv: "16c9b21550f875c96184bd0392670002"
server_key_version: "encryptionkey@0"
I20260811 03:10:01.478415 18237 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 03:10:01.482378 18251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:01.484113 18237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:10:01.484344 18237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/wal
uuid: "8437209937004cbea2fb31614d745052"
format_stamp: "Formatted at 2026-08-11 03:10:01 on dist-test-slave-8tdl"
server_key: "93da6f932e2d925221834b51923f48a7"
server_key_iv: "16c9b21550f875c96184bd0392670002"
server_key_version: "encryptionkey@0"
I20260811 03:10:01.484617 18237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:01.498970 18237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:01.499940 18237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:01.500278 18237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:01.501384 18237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 03:10:01.503172 18237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 03:10:01.503306 18237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:01.503445 18237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 03:10:01.503532 18237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:01.547997 18237 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.3:45573
I20260811 03:10:01.548094 18377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.3:45573 every 8 connection(s)
I20260811 03:10:01.549073 18092 heartbeater.cc:503] Master 127.27.109.62:34805 was elected leader, sending a full tablet report...
I20260811 03:10:01.549597 18237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/data/info.pb
I20260811 03:10:01.557569 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 18237
I20260811 03:10:01.557778 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/wal/instance
I20260811 03:10:01.558255 28084 external_mini_cluster.cc:1428] Setting key b9f045b90407b8780ba9617bb815628d
I20260811 03:10:01.564659 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.552552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.3:51429 (local address 127.27.109.62:34805)
0811 03:10:01.553022 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:01.553027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:01.554528 (+ 1501us) server_negotiation.cc:407] Connection header received
0811 03:10:01.556644 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:01.556650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:01.556776 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:01.556914 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:01.558314 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.559547 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.561279 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.561880 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.562056 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.563659 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:01.563664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:01.563668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:01.563687 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:01.563737 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:01.563750 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:01.563853 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:01.564101 (+ 248us) server_negotiation.cc:299] Negotiation successful
0811 03:10:01.564206 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":262,"threads_started":1}
I20260811 03:10:01.565760 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.553124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:34805 (local address 127.27.109.3:51429)
0811 03:10:01.554304 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:01.554385 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:01.556337 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:01.557210 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:01.557219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:01.557318 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:01.558105 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.558128 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.559732 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.559736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.561118 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.561131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.562170 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.562177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.562458 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.563284 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 03:10:01.563286 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:01.563334 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 03:10:01.563431 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:01.563855 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:01.563869 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:01.563880 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 03:10:01.564087 (+ 207us) client_negotiation.cc:240] Negotiation successful
0811 03:10:01.564547 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":404,"threads_started":1}
I20260811 03:10:01.567463 18378 heartbeater.cc:347] Connected to a master server at 127.27.109.62:34805
I20260811 03:10:01.567829 18378 heartbeater.cc:464] Registering TS with master...
I20260811 03:10:01.568681 18378 heartbeater.cc:511] Master 127.27.109.62:34805 requested a full tablet report, sending...
I20260811 03:10:01.570398 17892 ts_manager.cc:194] Registered new tserver with Master: 8437209937004cbea2fb31614d745052 (127.27.109.3:45573)
I20260811 03:10:01.571431 17892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.109.3:51429
I20260811 03:10:01.574810 28084 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 03:10:01.584961 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.579098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.27.109.62:34805)
0811 03:10:01.579290 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:01.579294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:01.579781 (+ 487us) server_negotiation.cc:407] Connection header received
0811 03:10:01.579926 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:01.579930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:01.579994 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:01.580111 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:01.580556 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.581287 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.582650 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.583105 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.583202 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.584315 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:01.584319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:01.584323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:01.584341 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:01.584374 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:01.584385 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:01.584452 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:01.584603 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 03:10:01.584683 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 03:10:01.589468 17892 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 03:10:01.596899 17892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 03:10:01.639060 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.610852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.2:36145 (local address 127.0.0.1:41844)
0811 03:10:01.628621 (+ 17769us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:01.628649 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:01.628823 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:01.630485 (+ 1662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:01.630490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:01.630522 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:10:01.631071 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.631088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.633061 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.633068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.635629 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.635647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.637739 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.637749 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.637981 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.638041 (+ 60us) client_negotiation.cc:769] Sending connection context
0811 03:10:01.638200 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 03:10:01.638489 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17572,"mutex_wait_us":4,"thread_start_us":19321,"threads_started":1}
I20260811 03:10:01.639235 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.610610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.0.0.1:55684)
0811 03:10:01.625170 (+ 14560us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:01.625238 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:01.625546 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:01.629707 (+ 4161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:01.629716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:01.629754 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:10:01.630282 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.630308 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.632595 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.632602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.635462 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.635487 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.638183 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.638190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.638425 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.638462 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 03:10:01.638561 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 03:10:01.638759 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14122,"mutex_wait_us":43,"thread_start_us":15212,"threads_started":1}
I20260811 03:10:01.639417 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.611489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.27.109.2:36145)
0811 03:10:01.629419 (+ 17930us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:01.629433 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:01.629474 (+ 41us) server_negotiation.cc:407] Connection header received
0811 03:10:01.629619 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:01.629631 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:01.629927 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:01.630186 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:10:01.631289 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.632896 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.635923 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.637495 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.637663 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.638045 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:01.638329 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 03:10:01.638525 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17738,"thread_start_us":17602,"threads_started":1}
I20260811 03:10:01.640384 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.610985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.27.109.1:37805)
0811 03:10:01.629048 (+ 18063us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:01.629056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:01.629088 (+ 32us) server_negotiation.cc:407] Connection header received
0811 03:10:01.629188 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:01.629195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:01.629430 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:01.629687 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:10:01.630573 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.632453 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.635773 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.637523 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.637662 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.637989 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:01.639768 (+ 1779us) server_negotiation.cc:299] Negotiation successful
0811 03:10:01.639978 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17861,"thread_start_us":17710,"threads_started":1}
I20260811 03:10:01.640712 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.612209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.27.109.3:45573)
0811 03:10:01.628065 (+ 15856us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:01.628075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:01.630901 (+ 2826us) server_negotiation.cc:407] Connection header received
0811 03:10:01.631031 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:01.631039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:01.631272 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:01.631488 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:10:01.632469 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.633811 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.636604 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.637692 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.637857 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.638163 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:01.639862 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0811 03:10:01.640093 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15618,"thread_start_us":15392,"threads_started":1}
I20260811 03:10:01.640950 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:01.611496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:45573 (local address 127.0.0.1:43512)
0811 03:10:01.630541 (+ 19045us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:01.630561 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:01.630737 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:01.631806 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:01.631812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:01.631840 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:10:01.632232 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.632244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.633994 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.634000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.636336 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:01.636408 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:01.638197 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:01.638203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:01.638832 (+ 629us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:01.638867 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 03:10:01.638982 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 03:10:01.639137 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18888,"mutex_wait_us":52,"thread_start_us":16750,"threads_started":1}
I20260811 03:10:01.644423 18164 tablet_service.cc:1467] Processing CreateTablet for tablet 155c408fe072416495149ba087f73106 (DEFAULT_TABLE table=test-table [id=a35b9c29496a490e90dba17e06bb4144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:10:01.644423 18026 tablet_service.cc:1467] Processing CreateTablet for tablet 155c408fe072416495149ba087f73106 (DEFAULT_TABLE table=test-table [id=a35b9c29496a490e90dba17e06bb4144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:10:01.644783 18305 tablet_service.cc:1467] Processing CreateTablet for tablet 155c408fe072416495149ba087f73106 (DEFAULT_TABLE table=test-table [id=a35b9c29496a490e90dba17e06bb4144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 03:10:01.646117 18026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155c408fe072416495149ba087f73106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:01.646119 18164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155c408fe072416495149ba087f73106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:01.646337 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 155c408fe072416495149ba087f73106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:01.660727 18397 tablet_bootstrap.cc:492] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Bootstrap starting.
I20260811 03:10:01.663614 18399 tablet_bootstrap.cc:492] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5: Bootstrap starting.
I20260811 03:10:01.665592 18397 tablet_bootstrap.cc:654] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:01.665741 18398 tablet_bootstrap.cc:492] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Bootstrap starting.
I20260811 03:10:01.667414 18397 log.cc:824] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:01.667863 18399 tablet_bootstrap.cc:654] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:01.669589 18399 log.cc:824] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:01.669822 18398 tablet_bootstrap.cc:654] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:01.671404 18398 log.cc:824] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:01.672407 18397 tablet_bootstrap.cc:492] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: No bootstrap required, opened a new log
I20260811 03:10:01.672749 18397 ts_tablet_manager.cc:1397] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260811 03:10:01.673370 18399 tablet_bootstrap.cc:492] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5: No bootstrap required, opened a new log
I20260811 03:10:01.673694 18399 ts_tablet_manager.cc:1397] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260811 03:10:01.674475 18398 tablet_bootstrap.cc:492] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: No bootstrap required, opened a new log
I20260811 03:10:01.674865 18398 ts_tablet_manager.cc:1397] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260811 03:10:01.684211 18398 raft_consensus.cc:348] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:01.684760 18398 raft_consensus.cc:374] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:01.684878 18398 raft_consensus.cc:729] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987937708c1548a79b5ca1411b8048f7, State: Initialized, Role: FOLLOWER
I20260811 03:10:01.685647 18398 consensus_queue.cc:260] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:01.685660 18397 raft_consensus.cc:348] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:01.686260 18397 raft_consensus.cc:374] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:01.686033 18399 raft_consensus.cc:348] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:01.686380 18397 raft_consensus.cc:729] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8437209937004cbea2fb31614d745052, State: Initialized, Role: FOLLOWER
I20260811 03:10:01.686537 18399 raft_consensus.cc:374] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:01.686661 18399 raft_consensus.cc:729] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be1179856ed4208bc8be1baa4f0adc5, State: Initialized, Role: FOLLOWER
I20260811 03:10:01.687255 18397 consensus_queue.cc:260] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:01.687299 18399 consensus_queue.cc:260] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:01.687511 18234 heartbeater.cc:503] Master 127.27.109.62:34805 was elected leader, sending a full tablet report...
I20260811 03:10:01.688089 18398 ts_tablet_manager.cc:1428] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260811 03:10:01.689363 18378 heartbeater.cc:503] Master 127.27.109.62:34805 was elected leader, sending a full tablet report...
I20260811 03:10:01.690918 18399 ts_tablet_manager.cc:1428] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5: Time spent starting tablet: real 0.017s user 0.014s sys 0.005s
I20260811 03:10:01.692049 18397 ts_tablet_manager.cc:1428] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
W20260811 03:10:01.776081 18093 tablet.cc:2369] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:10:01.786866 18235 tablet.cc:2369] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 03:10:01.802079 18379 tablet.cc:2369] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:10:02.037950 18403 raft_consensus.cc:482] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 03:10:02.038334 18403 raft_consensus.cc:504] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:02.040443 18403 leader_election.cc:288] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be1179856ed4208bc8be1baa4f0adc5 (127.27.109.1:37805), 8437209937004cbea2fb31614d745052 (127.27.109.3:45573)
I20260811 03:10:02.049623 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.041257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:60283 (local address 127.27.109.1:37805)
0811 03:10:02.041461 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:02.041467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:02.042224 (+ 757us) server_negotiation.cc:407] Connection header received
0811 03:10:02.042599 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:02.042605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:02.042724 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:02.042789 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:10:02.043773 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.044855 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.047383 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.048716 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.048875 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:02.048932 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:02.049270 (+ 338us) server_negotiation.cc:299] Negotiation successful
0811 03:10:02.049395 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 03:10:02.049616 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.041020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.109.2:59259 (local address 127.27.109.3:45573)
0811 03:10:02.041247 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:02.041254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:02.042449 (+ 1195us) server_negotiation.cc:407] Connection header received
0811 03:10:02.042601 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:02.042606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:02.042717 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:02.042811 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 03:10:02.043667 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.044446 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.047272 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.048533 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.048719 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:02.048814 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:02.049183 (+ 369us) server_negotiation.cc:299] Negotiation successful
0811 03:10:02.049337 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 03:10:02.049803 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.041437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.3:45573 (local address 127.27.109.2:59259)
0811 03:10:02.042313 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.042331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:02.042467 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:02.042846 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:02.042851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:02.042880 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:10:02.043444 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:02.043462 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.044601 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.044606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:02.047033 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:02.047057 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.048733 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.048738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:02.048993 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:02.049039 (+ 46us) client_negotiation.cc:769] Sending connection context
0811 03:10:02.049150 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 03:10:02.049303 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":135,"thread_start_us":568,"threads_started":1}
I20260811 03:10:02.049860 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.040865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.27.109.2:60283)
0811 03:10:02.042073 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.042102 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:02.042367 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:02.042997 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:02.043000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:02.043035 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 03:10:02.043529 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:02.043538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.045017 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.045023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:02.047183 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:02.047197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.048898 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.048904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:02.049108 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:02.049146 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 03:10:02.049223 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 03:10:02.049323 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":58,"thread_start_us":761,"threads_started":1}
I20260811 03:10:02.051354 18326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155c408fe072416495149ba087f73106" candidate_uuid: "987937708c1548a79b5ca1411b8048f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8437209937004cbea2fb31614d745052" is_pre_election: true
I20260811 03:10:02.051354 18046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155c408fe072416495149ba087f73106" candidate_uuid: "987937708c1548a79b5ca1411b8048f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be1179856ed4208bc8be1baa4f0adc5" is_pre_election: true
I20260811 03:10:02.052012 18326 raft_consensus.cc:2454] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 987937708c1548a79b5ca1411b8048f7 in term 0.
I20260811 03:10:02.052012 18046 raft_consensus.cc:2454] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 987937708c1548a79b5ca1411b8048f7 in term 0.
I20260811 03:10:02.052848 18112 leader_election.cc:302] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8437209937004cbea2fb31614d745052, 987937708c1548a79b5ca1411b8048f7; no voters:
I20260811 03:10:02.053326 18403 raft_consensus.cc:2781] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 03:10:02.053488 18403 raft_consensus.cc:482] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 03:10:02.053618 18403 raft_consensus.cc:3037] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:02.056082 18403 raft_consensus.cc:504] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:02.056795 18403 leader_election.cc:288] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [CANDIDATE]: Term 1 election: Requested vote from peers 0be1179856ed4208bc8be1baa4f0adc5 (127.27.109.1:37805), 8437209937004cbea2fb31614d745052 (127.27.109.3:45573)
I20260811 03:10:02.057453 18046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155c408fe072416495149ba087f73106" candidate_uuid: "987937708c1548a79b5ca1411b8048f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be1179856ed4208bc8be1baa4f0adc5"
I20260811 03:10:02.057453 18326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "155c408fe072416495149ba087f73106" candidate_uuid: "987937708c1548a79b5ca1411b8048f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8437209937004cbea2fb31614d745052"
I20260811 03:10:02.057706 18046 raft_consensus.cc:3037] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:02.057725 18326 raft_consensus.cc:3037] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:02.061218 18326 raft_consensus.cc:2454] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 987937708c1548a79b5ca1411b8048f7 in term 1.
I20260811 03:10:02.061218 18046 raft_consensus.cc:2454] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 987937708c1548a79b5ca1411b8048f7 in term 1.
I20260811 03:10:02.061803 18114 leader_election.cc:302] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0be1179856ed4208bc8be1baa4f0adc5, 8437209937004cbea2fb31614d745052, 987937708c1548a79b5ca1411b8048f7; no voters:
I20260811 03:10:02.062247 18403 raft_consensus.cc:2781] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:02.062814 18403 raft_consensus.cc:686] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 1 LEADER]: Becoming Leader. State: Replica: 987937708c1548a79b5ca1411b8048f7, State: Running, Role: LEADER
I20260811 03:10:02.063381 18403 consensus_queue.cc:237] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } }
I20260811 03:10:02.068478 17892 catalog_manager.cc:5495] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 987937708c1548a79b5ca1411b8048f7 (127.27.109.2). New cstate: current_term: 1 leader_uuid: "987937708c1548a79b5ca1411b8048f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "987937708c1548a79b5ca1411b8048f7" member_type: VOTER last_known_addr { host: "127.27.109.2" port: 36145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 } health_report { overall_health: UNKNOWN } } }
I20260811 03:10:02.103107 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.095975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.27.109.2:36145)
0811 03:10:02.096480 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:02.096488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:02.096828 (+ 340us) server_negotiation.cc:407] Connection header received
0811 03:10:02.097127 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:02.097131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:02.097246 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:02.097441 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:02.097911 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.098823 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.100557 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.101018 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.101182 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:02.101807 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:02.102401 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:02.102581 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:02.102719 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 03:10:02.102810 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260811 03:10:02.112393 18046 raft_consensus.cc:1261] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 1 FOLLOWER]: Refusing update from remote peer 987937708c1548a79b5ca1411b8048f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:10:02.112389 18326 raft_consensus.cc:1261] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 1 FOLLOWER]: Refusing update from remote peer 987937708c1548a79b5ca1411b8048f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:10:02.113462 18408 consensus_queue.cc:1035] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8437209937004cbea2fb31614d745052" member_type: VOTER last_known_addr { host: "127.27.109.3" port: 45573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:10:02.113863 18403 consensus_queue.cc:1035] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be1179856ed4208bc8be1baa4f0adc5" member_type: VOTER last_known_addr { host: "127.27.109.1" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:10:02.130657 18415 mvcc.cc:205] Tried to move back new op lower bound from 7317167317440716800 to 7317167317255610368. Current Snapshot: MvccSnapshot[applied={T|T < 7317167317440716800}]
I20260811 03:10:02.144418 17892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:10:02.144732 17892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 03:10:02.148960 17892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a640960e0fe45a4be16b54efd739f3a: Sending DeleteTablet for 3 replicas of tablet 155c408fe072416495149ba087f73106
I20260811 03:10:02.150352 18026 tablet_service.cc:1514] Processing DeleteTablet for tablet 155c408fe072416495149ba087f73106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:10:02 UTC) from {username='slave'} at 127.0.0.1:55684
I20260811 03:10:02.150514 18164 tablet_service.cc:1514] Processing DeleteTablet for tablet 155c408fe072416495149ba087f73106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:10:02 UTC) from {username='slave'} at 127.0.0.1:41844
I20260811 03:10:02.151630 18305 tablet_service.cc:1514] Processing DeleteTablet for tablet 155c408fe072416495149ba087f73106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 03:10:02 UTC) from {username='slave'} at 127.0.0.1:43512
I20260811 03:10:02.151823 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 17958
I20260811 03:10:02.174029 18425 tablet_replica.cc:330] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: stopping tablet replica
I20260811 03:10:02.174490 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.153432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.27.109.1:37805)
0811 03:10:02.153632 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:02.153640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:02.163516 (+ 9876us) server_negotiation.cc:407] Connection header received
0811 03:10:02.163749 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:02.163755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:02.163905 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:02.164167 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:02.164823 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.166373 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.169017 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.169948 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.170078 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:02.172380 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:02.172397 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:02.172423 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:02.173448 (+ 1025us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:02.173542 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:02.173578 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:02.173679 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:02.173957 (+ 278us) server_negotiation.cc:299] Negotiation successful
0811 03:10:02.174115 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 03:10:02.174914 18425 raft_consensus.cc:2229] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:10:02.175695 18424 tablet_replica.cc:330] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: stopping tablet replica
I20260811 03:10:02.175874 18086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:02.176436 18425 raft_consensus.cc:2258] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:10:02.177103 18424 raft_consensus.cc:2229] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 1 LEADER]: Raft consensus shutting down.
I20260811 03:10:02.179137 18424 raft_consensus.cc:2258] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:10:02.181217 18425 ts_tablet_manager.cc:1905] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:10:02.184455 18424 ts_tablet_manager.cc:1905] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 03:10:02.197340 18425 ts_tablet_manager.cc:1918] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:10:02.197597 18425 log.cc:1196] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-2/wal/wals/155c408fe072416495149ba087f73106
I20260811 03:10:02.198158 18425 ts_tablet_manager.cc:1939] T 155c408fe072416495149ba087f73106 P 8437209937004cbea2fb31614d745052: Deleting consensus metadata
I20260811 03:10:02.200218 17874 catalog_manager.cc:4841] TS 8437209937004cbea2fb31614d745052 (127.27.109.3:45573): tablet 155c408fe072416495149ba087f73106 (table test-table [id=a35b9c29496a490e90dba17e06bb4144]) successfully deleted
I20260811 03:10:02.204598 18424 ts_tablet_manager.cc:1918] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 03:10:02.204838 18424 log.cc:1196] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Deleting WAL directory at /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786417454123933-28084-0/minicluster-data/ts-1/wal/wals/155c408fe072416495149ba087f73106
I20260811 03:10:02.205384 18424 ts_tablet_manager.cc:1939] T 155c408fe072416495149ba087f73106 P 987937708c1548a79b5ca1411b8048f7: Deleting consensus metadata
I20260811 03:10:02.207180 17876 catalog_manager.cc:4841] TS 987937708c1548a79b5ca1411b8048f7 (127.27.109.2:36145): tablet 155c408fe072416495149ba087f73106 (table test-table [id=a35b9c29496a490e90dba17e06bb4144]) successfully deleted
I20260811 03:10:02.325430 18423 tablet_replica.cc:330] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5: stopping tablet replica
I20260811 03:10:02.325872 18423 raft_consensus.cc:2229] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 03:10:02.326328 18423 raft_consensus.cc:2258] T 155c408fe072416495149ba087f73106 P 0be1179856ed4208bc8be1baa4f0adc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 03:10:02.327333 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 17958
W20260811 03:10:02.349133 17877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.109.1:37805 (error 108)
I20260811 03:10:02.349273 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18096
I20260811 03:10:02.351446 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.350388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.0.0.1:55712)
0811 03:10:02.351051 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.351209 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":399,"threads_started":1}
W20260811 03:10:02.351702 17877 catalog_manager.cc:4584] TS 0be1179856ed4208bc8be1baa4f0adc5 (127.27.109.1:37805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 155c408fe072416495149ba087f73106: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37805: connect: Connection refused (error 111)
I20260811 03:10:02.358282 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.350394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.27.109.2:36145)
0811 03:10:02.350589 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:02.350595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:02.350619 (+ 24us) server_negotiation.cc:407] Connection header received
0811 03:10:02.350877 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:02.350882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:02.350990 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:02.351144 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:02.351794 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.352746 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.354179 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.355108 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.355253 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:02.356460 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:02.356474 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:02.356501 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:02.357373 (+ 872us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:02.357434 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:02.357469 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:02.357582 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:02.357724 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 03:10:02.357839 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 03:10:02.359352 18229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:02.405395 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.404875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.0.0.1:55728)
0811 03:10:02.405135 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.405230 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260811 03:10:02.406581 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.406214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.0.0.1:55734)
0811 03:10:02.406358 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.406434 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 03:10:02.467548 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.467058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.0.0.1:55748)
0811 03:10:02.467287 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.467378 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 03:10:02.468673 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.468267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.0.0.1:55764)
0811 03:10:02.468428 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.468508 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260811 03:10:02.485167 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 18096
I20260811 03:10:02.510753 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18237
I20260811 03:10:02.518429 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.511598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.27.109.3:45573)
0811 03:10:02.511818 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:02.511825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:02.511846 (+ 21us) server_negotiation.cc:407] Connection header received
0811 03:10:02.512102 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:02.512112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:02.512210 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:02.512319 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:02.512741 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.513706 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.515227 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:02.515651 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:02.515787 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:02.517012 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:02.517023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:02.517039 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:02.517624 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:02.517674 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:02.517695 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:02.517825 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:02.517961 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 03:10:02.518085 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 03:10:02.519162 18371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:02.549250 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.548730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.0.0.1:55776)
0811 03:10:02.548983 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.549085 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260811 03:10:02.550580 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:02.550053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.1:37805 (local address 127.0.0.1:55786)
0811 03:10:02.550296 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:02.550372 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.1:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260811 03:10:02.612607 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 18237
I20260811 03:10:02.632931 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 17858
I20260811 03:10:02.634299 17927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:02.750053 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 17858
2026-08-11T03:10:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3449 ms)
[----------] 6 tests from EncryptionPolicyTest (25165 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:02 dist-test-slave-8tdl krb5kdc[18436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:02 dist-test-slave-8tdl krb5kdc[18436](info): set up 2 sockets
Aug 11 03:10:02 dist-test-slave-8tdl krb5kdc[18436](info): commencing operation
krb5kdc: starting...
W20260811 03:10:04.049738 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:04 dist-test-slave-8tdl krb5kdc[18436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:04.414953 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:04.784821 18451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:04.785168 18451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:04.785303 18451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:04.793494 18451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:04.793638 18451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:04.793731 18451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:04.793831 18451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:04.793921 18451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:04.804663 18451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:04.806210 18451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:04.807732 18451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:04.820511 18459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:04.820575 18457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:04.820799 18456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:04.821827 18451 server_base.cc:1034] running on GCE node
I20260811 03:10:04.822865 18451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:04.824422 18451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:04.825644 18451 hybrid_clock.cc:648] HybridClock initialized: now 1786417804825598 us; error 34 us; skew 500 ppm
I20260811 03:10:04.825978 18451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:04 dist-test-slave-8tdl krb5kdc[18436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:04.829656 18451 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:04.831534 18451 webserver.cc:460] Webserver started at http://127.0.0.1:46123/ using document root <none> and password file <none>
I20260811 03:10:04.832306 18451 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:04.832464 18451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:04.832801 18451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:04.836560 18451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "f0d986c85df448c39a10de4f2ef5a0d5"
format_stamp: "Formatted at 2026-08-11 03:10:04 on dist-test-slave-8tdl"
server_key: "b99b0bebf810157a1f0b9e447bae4910"
server_key_iv: "5a5be52dfafb618afed4f54437b00f5a"
server_key_version: "encryptionkey@0"
I20260811 03:10:04.837574 18451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "f0d986c85df448c39a10de4f2ef5a0d5"
format_stamp: "Formatted at 2026-08-11 03:10:04 on dist-test-slave-8tdl"
server_key: "b99b0bebf810157a1f0b9e447bae4910"
server_key_iv: "5a5be52dfafb618afed4f54437b00f5a"
server_key_version: "encryptionkey@0"
I20260811 03:10:04.843664 18451 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260811 03:10:04.847630 18468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:04.849198 18451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:10:04.849434 18451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "f0d986c85df448c39a10de4f2ef5a0d5"
format_stamp: "Formatted at 2026-08-11 03:10:04 on dist-test-slave-8tdl"
server_key: "b99b0bebf810157a1f0b9e447bae4910"
server_key_iv: "5a5be52dfafb618afed4f54437b00f5a"
server_key_version: "encryptionkey@0"
I20260811 03:10:04.849686 18451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:04.871423 18451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:04.876257 18451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:04.876590 18451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:04.897409 18451 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34487
I20260811 03:10:04.897454 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34487 every 8 connection(s)
I20260811 03:10:04.898906 18451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:04.901069 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 18451
I20260811 03:10:04.901324 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:04.902383 28084 external_mini_cluster.cc:1428] Setting key 93b121c1d23a3f503521b46e5184633a
I20260811 03:10:04.904243 18526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 03:10:04 dist-test-slave-8tdl krb5kdc[18436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 03:10:04.918241 18526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5: Bootstrap starting.
I20260811 03:10:04.922317 18526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:04.923538 18526 log.cc:824] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:04.926891 18526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5: No bootstrap required, opened a new log
I20260811 03:10:04.927085 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:04.904642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48493 (local address 127.0.0.1:34487)
0811 03:10:04.905578 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:04.905590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:04.905636 (+ 46us) server_negotiation.cc:407] Connection header received
0811 03:10:04.907458 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:04.907505 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:04.907974 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:04.908393 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:04.909945 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:04.911430 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:04.912746 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:04.913456 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:04.913658 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:04.918588 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:04.918596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:04.918611 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:04.918651 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:04.922165 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:04.923206 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:04.923214 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:04.923221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:04.923424 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:04.924036 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:04.924040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:04.924042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:04.924645 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:04.925002 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:04.925538 (+ 536us) server_negotiation.cc:299] Negotiation successful
0811 03:10:04.925908 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":415,"threads_started":1}
I20260811 03:10:04.931484 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:04.935779 18526 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d986c85df448c39a10de4f2ef5a0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34487 } }
I20260811 03:10:04.936334 18526 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:04.936481 18526 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d986c85df448c39a10de4f2ef5a0d5, State: Initialized, Role: FOLLOWER
I20260811 03:10:04.937372 18526 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d986c85df448c39a10de4f2ef5a0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34487 } }
I20260811 03:10:04.937744 18526 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:04.937923 18526 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:04.938181 18526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:04.940932 18526 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d986c85df448c39a10de4f2ef5a0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34487 } }
I20260811 03:10:04.941638 18526 leader_election.cc:302] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0d986c85df448c39a10de4f2ef5a0d5; no voters:
I20260811 03:10:04.942401 18526 leader_election.cc:288] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:04.942559 18537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:04.943825 18537 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [term 1 LEADER]: Becoming Leader. State: Replica: f0d986c85df448c39a10de4f2ef5a0d5, State: Running, Role: LEADER
I20260811 03:10:04.944612 18537 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d986c85df448c39a10de4f2ef5a0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34487 } }
I20260811 03:10:04.945026 18526 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:04.950117 18538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0d986c85df448c39a10de4f2ef5a0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d986c85df448c39a10de4f2ef5a0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34487 } } }
I20260811 03:10:04.950610 18538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:04.960901 18539 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d986c85df448c39a10de4f2ef5a0d5. Latest consensus state: current_term: 1 leader_uuid: "f0d986c85df448c39a10de4f2ef5a0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d986c85df448c39a10de4f2ef5a0d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34487 } } }
I20260811 03:10:04.961412 18539 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:04.961900 18546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:04.962278 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:04.936523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45749 (local address 127.0.0.1:34487)
0811 03:10:04.936682 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:04.936687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:04.939263 (+ 2576us) server_negotiation.cc:407] Connection header received
0811 03:10:04.939493 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:04.939497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:04.939588 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:04.939714 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:04.941218 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:04.942492 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:04.944696 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:04.945417 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:04.945584 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:04.949733 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:04.949743 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:04.949749 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:04.949807 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:04.954918 (+ 5111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:04.957041 (+ 2123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:04.957050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:04.957056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:04.957346 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:04.958219 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:04.958227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:04.958233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:04.958848 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:04.960769 (+ 1921us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:04.961001 (+ 232us) server_negotiation.cc:299] Negotiation successful
0811 03:10:04.961238 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 03:10:04.967906 18546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:04.977946 18546 catalog_manager.cc:1347] Generated new cluster ID: 98ec94140572457b93bd1ae00594f959
I20260811 03:10:04.978093 18546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:05.007764 18546 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:05.009840 18546 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:05.017719 18546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0d986c85df448c39a10de4f2ef5a0d5: Generated new TSK 0
I20260811 03:10:05.018827 18546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:05.026149 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18451
I20260811 03:10:05.028366 18521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:05.125583 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 18451
W20260811 03:10:05.148103 18433 connection.cc:535] client connection to 127.0.0.1:34487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34487): Connection reset by peer (error 104)
2026-08-11T03:10:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:05 dist-test-slave-8tdl krb5kdc[18561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:05 dist-test-slave-8tdl krb5kdc[18561](info): set up 2 sockets
Aug 11 03:10:05 dist-test-slave-8tdl krb5kdc[18561](info): commencing operation
krb5kdc: starting...
W20260811 03:10:06.366148 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.002s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:06 dist-test-slave-8tdl krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:06.728571 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:07.103643 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:07.103966 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:07.104175 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:07.111886 18574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:07.112030 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:07.112116 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:07.112227 18574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:07.112313 18574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:07.123157 18574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:07.124622 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:07.126034 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:07.138531 18581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:07.138562 18582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:07.138600 18584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:07.139991 18574 server_base.cc:1034] running on GCE node
I20260811 03:10:07.140954 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:07.142927 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:07.144204 18574 hybrid_clock.cc:648] HybridClock initialized: now 1786417807144154 us; error 37 us; skew 500 ppm
I20260811 03:10:07.144593 18574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:07 dist-test-slave-8tdl krb5kdc[18561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:07.153707 18574 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:07.155824 18574 webserver.cc:460] Webserver started at http://127.0.0.1:46585/ using document root <none> and password file <none>
I20260811 03:10:07.156687 18574 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:07.156872 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:07.157297 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:07.161079 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "b249d0bb82b54c90a957adeb39b01f73"
format_stamp: "Formatted at 2026-08-11 03:10:07 on dist-test-slave-8tdl"
server_key: "b19aba6cc7bdf70dac9446a2f173e005"
server_key_iv: "6ea75c5262d9f531411dd7995431dfc6"
server_key_version: "encryptionkey@0"
I20260811 03:10:07.162113 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "b249d0bb82b54c90a957adeb39b01f73"
format_stamp: "Formatted at 2026-08-11 03:10:07 on dist-test-slave-8tdl"
server_key: "b19aba6cc7bdf70dac9446a2f173e005"
server_key_iv: "6ea75c5262d9f531411dd7995431dfc6"
server_key_version: "encryptionkey@0"
I20260811 03:10:07.168653 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260811 03:10:07.173134 18592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:07.174930 18574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 03:10:07.175201 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "b249d0bb82b54c90a957adeb39b01f73"
format_stamp: "Formatted at 2026-08-11 03:10:07 on dist-test-slave-8tdl"
server_key: "b19aba6cc7bdf70dac9446a2f173e005"
server_key_iv: "6ea75c5262d9f531411dd7995431dfc6"
server_key_version: "encryptionkey@0"
I20260811 03:10:07.175442 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:07.209370 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:07.214308 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:07.214628 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:07.235271 18574 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46611
I20260811 03:10:07.235303 18647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46611 every 8 connection(s)
I20260811 03:10:07.236963 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:07.241225 18648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:07.246762 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 18574
I20260811 03:10:07.246974 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:07.247489 28084 external_mini_cluster.cc:1428] Setting key 9bb09046ed97dd2786be6c88db59ca2f
I20260811 03:10:07.252863 18648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73: Bootstrap starting.
I20260811 03:10:07.256985 18648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:07.258277 18648 log.cc:824] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:07.261772 18648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73: No bootstrap required, opened a new log
I20260811 03:10:07.262669 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:07.249608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40041 (local address 127.0.0.1:46611)
0811 03:10:07.250564 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:07.250577 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:07.250626 (+ 49us) server_negotiation.cc:407] Connection header received
0811 03:10:07.252433 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:07.252459 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:07.252903 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:07.253313 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:07.254186 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:07.255676 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:07.257694 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:07.258429 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:07.258649 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:07.260078 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:07.260086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:07.260102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:07.260675 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:07.260761 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:07.260788 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:07.260971 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:07.261258 (+ 287us) server_negotiation.cc:299] Negotiation successful
0811 03:10:07.261455 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":439,"threads_started":1}
I20260811 03:10:07.267402 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:07.271658 18648 raft_consensus.cc:348] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b249d0bb82b54c90a957adeb39b01f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46611 } }
I20260811 03:10:07.272187 18648 raft_consensus.cc:374] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:07.272328 18648 raft_consensus.cc:729] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b249d0bb82b54c90a957adeb39b01f73, State: Initialized, Role: FOLLOWER
I20260811 03:10:07.273147 18648 consensus_queue.cc:260] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b249d0bb82b54c90a957adeb39b01f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46611 } }
I20260811 03:10:07.273488 18648 raft_consensus.cc:388] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:07.273669 18648 raft_consensus.cc:482] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:07.273857 18648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:07.277082 18648 raft_consensus.cc:504] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b249d0bb82b54c90a957adeb39b01f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46611 } }
I20260811 03:10:07.277850 18648 leader_election.cc:302] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b249d0bb82b54c90a957adeb39b01f73; no voters:
I20260811 03:10:07.278647 18648 leader_election.cc:288] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:07.278900 18658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:07.279728 18658 raft_consensus.cc:686] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [term 1 LEADER]: Becoming Leader. State: Replica: b249d0bb82b54c90a957adeb39b01f73, State: Running, Role: LEADER
I20260811 03:10:07.280753 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:07.272074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43975 (local address 127.0.0.1:46611)
0811 03:10:07.272309 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:07.272320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:07.272846 (+ 526us) server_negotiation.cc:407] Connection header received
0811 03:10:07.273050 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:07.273056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:07.273163 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:07.273296 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:07.273938 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:07.275226 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:07.276681 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:07.277470 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:07.277614 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:07.279350 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:07.279357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:07.279363 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:07.279396 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:07.279469 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:07.279510 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:07.279669 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:07.280302 (+ 633us) server_negotiation.cc:299] Negotiation successful
0811 03:10:07.280370 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260811 03:10:07.280830 18658 consensus_queue.cc:237] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b249d0bb82b54c90a957adeb39b01f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46611 } }
I20260811 03:10:07.283915 18659 sys_catalog.cc:455] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b249d0bb82b54c90a957adeb39b01f73. Latest consensus state: current_term: 1 leader_uuid: "b249d0bb82b54c90a957adeb39b01f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b249d0bb82b54c90a957adeb39b01f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46611 } } }
I20260811 03:10:07.284435 18659 sys_catalog.cc:458] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:07.284890 18660 sys_catalog.cc:455] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b249d0bb82b54c90a957adeb39b01f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b249d0bb82b54c90a957adeb39b01f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46611 } } }
I20260811 03:10:07.285315 18660 sys_catalog.cc:458] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:07.285501 18648 sys_catalog.cc:564] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:07.287813 18662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:07.293823 18662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:07.304023 18662 catalog_manager.cc:1347] Generated new cluster ID: 0ebcd8f096c54e11afbf592bd15aa9fb
I20260811 03:10:07.304174 18662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:07.322077 18662 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:07.324123 18662 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:07.331142 18662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b249d0bb82b54c90a957adeb39b01f73: Generated new TSK 0
I20260811 03:10:07.332459 18662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:07.352178 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18574
W20260811 03:10:07.353660 18642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40041
W20260811 03:10:07.354723 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:07.355048 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 18574
W20260811 03:10:07.380198 18558 connection.cc:535] client connection to 127.0.0.1:46611 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46611): Connection reset by peer (error 104)
2026-08-11T03:10:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:07 dist-test-slave-8tdl krb5kdc[18681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:07 dist-test-slave-8tdl krb5kdc[18681](info): set up 2 sockets
Aug 11 03:10:07 dist-test-slave-8tdl krb5kdc[18681](info): commencing operation
krb5kdc: starting...
W20260811 03:10:08.603226 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:08 dist-test-slave-8tdl krb5kdc[18681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:08.991334 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:09.360000 18695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:09.360340 18695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:09.360513 18695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:09.371582 18695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:09.371708 18695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:09.371805 18695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:09.371917 18695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:09.372004 18695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:09.383419 18695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:09.384990 18695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:09.386435 18695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:09.399612 18701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:09.399523 18704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:09.399566 18700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:09.400633 18695 server_base.cc:1034] running on GCE node
I20260811 03:10:09.401546 18695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:09.403250 18695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:09.404531 18695 hybrid_clock.cc:648] HybridClock initialized: now 1786417809404480 us; error 42 us; skew 500 ppm
I20260811 03:10:09.404965 18695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:09 dist-test-slave-8tdl krb5kdc[18681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:09.408496 18695 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:09.410153 18695 webserver.cc:460] Webserver started at http://127.0.0.1:43297/ using document root <none> and password file <none>
I20260811 03:10:09.410917 18695 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:09.411037 18695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:09.411332 18695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:09.414511 18695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "c8556683108941018588f3ab36696706"
format_stamp: "Formatted at 2026-08-11 03:10:09 on dist-test-slave-8tdl"
server_key: "e6ec4c042230e6f73f889cf3af9c21ef"
server_key_iv: "9ea49e0945f18127fc9fbc6493763d64"
server_key_version: "encryptionkey@0"
I20260811 03:10:09.415462 18695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "c8556683108941018588f3ab36696706"
format_stamp: "Formatted at 2026-08-11 03:10:09 on dist-test-slave-8tdl"
server_key: "e6ec4c042230e6f73f889cf3af9c21ef"
server_key_iv: "9ea49e0945f18127fc9fbc6493763d64"
server_key_version: "encryptionkey@0"
I20260811 03:10:09.421025 18695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 03:10:09.424701 18711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:09.426163 18695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:10:09.426366 18695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "c8556683108941018588f3ab36696706"
format_stamp: "Formatted at 2026-08-11 03:10:09 on dist-test-slave-8tdl"
server_key: "e6ec4c042230e6f73f889cf3af9c21ef"
server_key_iv: "9ea49e0945f18127fc9fbc6493763d64"
server_key_version: "encryptionkey@0"
I20260811 03:10:09.426590 18695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:09.432651 18695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:09.437278 18695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:09.437654 18695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:09.457602 18695 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46317
I20260811 03:10:09.457659 18765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46317 every 8 connection(s)
I20260811 03:10:09.459234 18695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:09.460153 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 18695
I20260811 03:10:09.460388 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:09.460876 28084 external_mini_cluster.cc:1428] Setting key ccc6662e081accdd15a2b6d985b60bc5
I20260811 03:10:09.464501 18766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:09.469570 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:09.462638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55143 (local address 127.0.0.1:46317)
0811 03:10:09.463604 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:09.463621 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:09.463955 (+ 334us) server_negotiation.cc:407] Connection header received
0811 03:10:09.465942 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:09.465965 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:09.466292 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:09.466682 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:09.467850 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:09.467856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:09.467869 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:09.467888 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:09.467964 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:09.467984 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:09.468250 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:09.468426 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 03:10:09.468609 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":381,"threads_started":1}
I20260811 03:10:09.473984 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:09.477313 18766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706: Bootstrap starting.
I20260811 03:10:09.481444 18766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:09.482153 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:09.479104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55283 (local address 127.0.0.1:46317)
0811 03:10:09.479337 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:09.479346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:09.479733 (+ 387us) server_negotiation.cc:407] Connection header received
0811 03:10:09.479970 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:09.479974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:09.480029 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:09.480138 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:09.481286 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:09.481293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:09.481305 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:09.481328 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:09.481374 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:09.481391 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:09.481534 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:09.481710 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 03:10:09.481781 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 03:10:09.483116 18766 log.cc:824] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:09.486232 18766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706: No bootstrap required, opened a new log
I20260811 03:10:09.494248 18766 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8556683108941018588f3ab36696706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } }
I20260811 03:10:09.494721 18766 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:09.494828 18766 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8556683108941018588f3ab36696706, State: Initialized, Role: FOLLOWER
I20260811 03:10:09.495652 18766 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8556683108941018588f3ab36696706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } }
I20260811 03:10:09.495960 18766 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:09.496147 18766 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:09.496344 18766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:09.498647 18766 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8556683108941018588f3ab36696706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } }
I20260811 03:10:09.499289 18766 leader_election.cc:302] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8556683108941018588f3ab36696706; no voters:
I20260811 03:10:09.499969 18766 leader_election.cc:288] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:09.500137 18776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:09.501446 18776 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [term 1 LEADER]: Becoming Leader. State: Replica: c8556683108941018588f3ab36696706, State: Running, Role: LEADER
I20260811 03:10:09.502161 18776 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8556683108941018588f3ab36696706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } }
I20260811 03:10:09.502301 18766 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:09.505338 18778 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8556683108941018588f3ab36696706. Latest consensus state: current_term: 1 leader_uuid: "c8556683108941018588f3ab36696706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8556683108941018588f3ab36696706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } } }
I20260811 03:10:09.506362 18778 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:09.505430 18777 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8556683108941018588f3ab36696706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8556683108941018588f3ab36696706" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46317 } } }
I20260811 03:10:09.506531 18777 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:09.507589 18786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:09.513808 18786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:09.524416 18786 catalog_manager.cc:1347] Generated new cluster ID: a090e2ca68fc4f028aa5c3cd132bb601
I20260811 03:10:09.524605 18786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:09.539963 18786 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:09.540443 18786 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:09.553462 18786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8556683108941018588f3ab36696706: Generated new TSK 0
I20260811 03:10:09.554481 18786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:09.566201 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18695
W20260811 03:10:09.567564 18761 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55143
W20260811 03:10:09.568531 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18695: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:09.568756 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 18695
2026-08-11T03:10:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:09 dist-test-slave-8tdl krb5kdc[18798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:09 dist-test-slave-8tdl krb5kdc[18798](info): set up 2 sockets
Aug 11 03:10:09 dist-test-slave-8tdl krb5kdc[18798](info): commencing operation
krb5kdc: starting...
W20260811 03:10:10.818293 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:11 dist-test-slave-8tdl krb5kdc[18798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:11.181741 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:11.547608 18811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:11.547951 18811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:11.548053 18811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:11.555819 18811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:11.555958 18811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:11.556007 18811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:11.556071 18811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:11.556118 18811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:11.567214 18811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:11.568707 18811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:11.570120 18811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:11.583364 18816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:11.583459 18817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:11.583413 18819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:11.584645 18811 server_base.cc:1034] running on GCE node
I20260811 03:10:11.585526 18811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:11.587208 18811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:11.588425 18811 hybrid_clock.cc:648] HybridClock initialized: now 1786417811588384 us; error 39 us; skew 500 ppm
I20260811 03:10:11.588819 18811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:11 dist-test-slave-8tdl krb5kdc[18798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:11.592413 18811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:11.594174 18811 webserver.cc:460] Webserver started at http://127.0.0.1:36569/ using document root <none> and password file <none>
I20260811 03:10:11.595006 18811 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:11.595167 18811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:11.595492 18811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:11.598658 18811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "d176f2898272469d990706cac40c90e0"
format_stamp: "Formatted at 2026-08-11 03:10:11 on dist-test-slave-8tdl"
server_key: "d626a29b29d2eb6bb25ab21de8b61cac"
server_key_iv: "2ae0b4714a9d4e65ed895f652a1f2f9d"
server_key_version: "encryptionkey@0"
I20260811 03:10:11.599555 18811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "d176f2898272469d990706cac40c90e0"
format_stamp: "Formatted at 2026-08-11 03:10:11 on dist-test-slave-8tdl"
server_key: "d626a29b29d2eb6bb25ab21de8b61cac"
server_key_iv: "2ae0b4714a9d4e65ed895f652a1f2f9d"
server_key_version: "encryptionkey@0"
I20260811 03:10:11.605252 18811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:10:11.609181 18827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:11.610801 18811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:10:11.611043 18811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "d176f2898272469d990706cac40c90e0"
format_stamp: "Formatted at 2026-08-11 03:10:11 on dist-test-slave-8tdl"
server_key: "d626a29b29d2eb6bb25ab21de8b61cac"
server_key_iv: "2ae0b4714a9d4e65ed895f652a1f2f9d"
server_key_version: "encryptionkey@0"
I20260811 03:10:11.611291 18811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:11.626955 18811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:11.631976 18811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:11.632328 18811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:11.655283 18811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46111
I20260811 03:10:11.655334 18884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46111 every 8 connection(s)
I20260811 03:10:11.656885 18811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:11.657285 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 18811
I20260811 03:10:11.657485 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:11.658205 28084 external_mini_cluster.cc:1428] Setting key fc0c88b103f8c14198709837c29c3686
I20260811 03:10:11.661614 18885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 03:10:11 dist-test-slave-8tdl krb5kdc[18798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 03:10:11.675928 18885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0: Bootstrap starting.
I20260811 03:10:11.679392 18885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:11.680619 18885 log.cc:824] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:11.683099 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:11.660296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55989 (local address 127.0.0.1:46111)
0811 03:10:11.661101 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:11.661113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:11.661154 (+ 41us) server_negotiation.cc:407] Connection header received
0811 03:10:11.662950 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:11.662972 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:11.663351 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:11.663672 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:11.665632 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:11.667160 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:11.669034 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:11.669771 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:11.669910 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:11.675318 (+ 5408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:11.675327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:11.675344 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:11.675386 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:11.678571 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:11.679448 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:11.679456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:11.679463 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:11.679693 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:11.680126 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:11.680133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:11.680137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:11.680711 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:11.681036 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:11.681548 (+ 512us) server_negotiation.cc:299] Negotiation successful
0811 03:10:11.682001 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":379,"threads_started":1}
I20260811 03:10:11.683981 18885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0: No bootstrap required, opened a new log
I20260811 03:10:11.687101 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:11.692646 18885 raft_consensus.cc:348] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d176f2898272469d990706cac40c90e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } }
I20260811 03:10:11.693202 18885 raft_consensus.cc:374] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:11.693316 18885 raft_consensus.cc:729] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d176f2898272469d990706cac40c90e0, State: Initialized, Role: FOLLOWER
I20260811 03:10:11.693980 18885 consensus_queue.cc:260] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d176f2898272469d990706cac40c90e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } }
I20260811 03:10:11.694232 18885 raft_consensus.cc:388] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:11.694414 18885 raft_consensus.cc:482] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:11.694614 18885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:11.697073 18885 raft_consensus.cc:504] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d176f2898272469d990706cac40c90e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } }
I20260811 03:10:11.697764 18885 leader_election.cc:302] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d176f2898272469d990706cac40c90e0; no voters:
I20260811 03:10:11.698571 18885 leader_election.cc:288] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:11.698730 18896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:11.700910 18896 raft_consensus.cc:686] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [term 1 LEADER]: Becoming Leader. State: Replica: d176f2898272469d990706cac40c90e0, State: Running, Role: LEADER
I20260811 03:10:11.701216 18885 sys_catalog.cc:564] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:11.701658 18896 consensus_queue.cc:237] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d176f2898272469d990706cac40c90e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } }
I20260811 03:10:11.708228 18898 sys_catalog.cc:455] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d176f2898272469d990706cac40c90e0. Latest consensus state: current_term: 1 leader_uuid: "d176f2898272469d990706cac40c90e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d176f2898272469d990706cac40c90e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } } }
I20260811 03:10:11.708655 18898 sys_catalog.cc:458] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:11.710353 18897 sys_catalog.cc:455] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d176f2898272469d990706cac40c90e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d176f2898272469d990706cac40c90e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46111 } } }
I20260811 03:10:11.710875 18897 sys_catalog.cc:458] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:11.711580 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:11.691831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57443 (local address 127.0.0.1:46111)
0811 03:10:11.692033 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:11.692040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:11.692400 (+ 360us) server_negotiation.cc:407] Connection header received
0811 03:10:11.692544 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:11.692549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:11.692646 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:11.692766 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:11.694583 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:11.695463 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:11.696746 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:11.697217 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:11.697366 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:11.700775 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:11.700785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:11.700793 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:11.700847 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:11.705591 (+ 4744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:11.706929 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:11.706942 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:11.706948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:11.707292 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:11.707934 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:11.707942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:11.707949 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:11.708494 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:11.710381 (+ 1887us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:11.710581 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 03:10:11.710969 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 03:10:11.714994 18904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:11.721143 18904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:11.732862 18904 catalog_manager.cc:1347] Generated new cluster ID: e2d848d029d74eb8b371aa0cfb25bb40
I20260811 03:10:11.733004 18904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:11.769018 18904 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:11.770879 18904 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:11.778501 18904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d176f2898272469d990706cac40c90e0: Generated new TSK 0
I20260811 03:10:11.779415 18904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:11.793856 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18811
I20260811 03:10:11.795205 18880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:11.894968 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 18811
2026-08-11T03:10:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:12 dist-test-slave-8tdl krb5kdc[18918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:12 dist-test-slave-8tdl krb5kdc[18918](info): set up 2 sockets
Aug 11 03:10:12 dist-test-slave-8tdl krb5kdc[18918](info): commencing operation
krb5kdc: starting...
W20260811 03:10:13.106392 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:13 dist-test-slave-8tdl krb5kdc[18918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:13.469858 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:13.845466 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:13.845790 18931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:13.845897 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:13.853549 18931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:13.853688 18931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:13.853737 18931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:13.853794 18931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:13.853840 18931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:13.864689 18931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:13.866185 18931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:13.867708 18931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:13.880693 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:13.880671 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:13.880615 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:13.881939 18931 server_base.cc:1034] running on GCE node
I20260811 03:10:13.882944 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:13.884635 18931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:13.885910 18931 hybrid_clock.cc:648] HybridClock initialized: now 1786417813885885 us; error 51 us; skew 500 ppm
I20260811 03:10:13.886336 18931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:13 dist-test-slave-8tdl krb5kdc[18918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:13.894098 18931 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:13.896303 18931 webserver.cc:460] Webserver started at http://127.0.0.1:46411/ using document root <none> and password file <none>
I20260811 03:10:13.897148 18931 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:13.897328 18931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:13.897708 18931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:13.901247 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "d889ac02331b4b859323137060191090"
format_stamp: "Formatted at 2026-08-11 03:10:13 on dist-test-slave-8tdl"
server_key: "e0efd310f661321931152e3fe07027ef"
server_key_iv: "fa9c477f82a077b7c51ee5855b27f489"
server_key_version: "encryptionkey@0"
I20260811 03:10:13.902186 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "d889ac02331b4b859323137060191090"
format_stamp: "Formatted at 2026-08-11 03:10:13 on dist-test-slave-8tdl"
server_key: "e0efd310f661321931152e3fe07027ef"
server_key_iv: "fa9c477f82a077b7c51ee5855b27f489"
server_key_version: "encryptionkey@0"
I20260811 03:10:13.908399 18931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260811 03:10:13.912539 18949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:13.914137 18931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 03:10:13.914453 18931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "d889ac02331b4b859323137060191090"
format_stamp: "Formatted at 2026-08-11 03:10:13 on dist-test-slave-8tdl"
server_key: "e0efd310f661321931152e3fe07027ef"
server_key_iv: "fa9c477f82a077b7c51ee5855b27f489"
server_key_version: "encryptionkey@0"
I20260811 03:10:13.914743 18931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:13.946595 18931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:13.951642 18931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:13.951974 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:13.972973 18931 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45317
I20260811 03:10:13.973037 19003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45317 every 8 connection(s)
I20260811 03:10:13.974514 18931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:13.977984 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 18931
I20260811 03:10:13.978222 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:13.978819 28084 external_mini_cluster.cc:1428] Setting key cac5f93adc4b18331b3f0415ca5a0dc5
I20260811 03:10:13.979429 19004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:13.992909 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:13.980769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38501 (local address 127.0.0.1:45317)
0811 03:10:13.981606 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:13.981623 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:13.981666 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:10:13.983366 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:13.983396 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:13.983840 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:13.984279 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:13.984884 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:13.986310 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:13.987850 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:13.988778 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:13.989119 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:13.990352 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:13.990360 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:13.990378 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:13.991092 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:13.991174 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:13.991208 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:13.991349 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:13.991553 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 03:10:13.991870 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":378,"threads_started":1}
I20260811 03:10:13.992903 19004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090: Bootstrap starting.
I20260811 03:10:13.997076 19004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:13.997676 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:13.998431 19004 log.cc:824] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:14.001781 19004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090: No bootstrap required, opened a new log
I20260811 03:10:14.012315 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:14.002908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35959 (local address 127.0.0.1:45317)
0811 03:10:14.003168 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:14.003176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:14.003884 (+ 708us) server_negotiation.cc:407] Connection header received
0811 03:10:14.004095 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:14.004100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:14.004173 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:14.004272 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:14.004966 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:14.005759 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:14.008331 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:14.008889 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:14.009091 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:14.010999 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:14.011008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:14.011011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:14.011053 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:14.011123 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:14.011148 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:14.011243 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:14.011487 (+ 244us) server_negotiation.cc:299] Negotiation successful
0811 03:10:14.011903 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 03:10:14.014176 19004 raft_consensus.cc:348] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889ac02331b4b859323137060191090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } }
I20260811 03:10:14.014787 19004 raft_consensus.cc:374] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:14.014930 19004 raft_consensus.cc:729] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d889ac02331b4b859323137060191090, State: Initialized, Role: FOLLOWER
I20260811 03:10:14.015595 19004 consensus_queue.cc:260] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889ac02331b4b859323137060191090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } }
I20260811 03:10:14.015903 19004 raft_consensus.cc:388] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:14.016074 19004 raft_consensus.cc:482] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:14.016271 19004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:14.020678 19004 raft_consensus.cc:504] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889ac02331b4b859323137060191090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } }
I20260811 03:10:14.021440 19004 leader_election.cc:302] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d889ac02331b4b859323137060191090; no voters:
I20260811 03:10:14.022167 19004 leader_election.cc:288] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:14.022367 19014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:14.023299 19014 raft_consensus.cc:686] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [term 1 LEADER]: Becoming Leader. State: Replica: d889ac02331b4b859323137060191090, State: Running, Role: LEADER
I20260811 03:10:14.023908 19014 consensus_queue.cc:237] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889ac02331b4b859323137060191090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } }
I20260811 03:10:14.024753 19004 sys_catalog.cc:564] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:14.027999 19016 sys_catalog.cc:455] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d889ac02331b4b859323137060191090. Latest consensus state: current_term: 1 leader_uuid: "d889ac02331b4b859323137060191090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889ac02331b4b859323137060191090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } } }
I20260811 03:10:14.028555 19016 sys_catalog.cc:458] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:14.029579 19015 sys_catalog.cc:455] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d889ac02331b4b859323137060191090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d889ac02331b4b859323137060191090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45317 } } }
I20260811 03:10:14.030107 19015 sys_catalog.cc:458] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:14.030656 19020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:14.036559 19020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:14.047719 19020 catalog_manager.cc:1347] Generated new cluster ID: ae45bd331b9e4bbea596bea610e3be3a
I20260811 03:10:14.047955 19020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:14.063252 19020 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:14.065387 19020 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:14.087178 19020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d889ac02331b4b859323137060191090: Generated new TSK 0
I20260811 03:10:14.088506 19020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:14.109876 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18931
W20260811 03:10:14.111656 18999 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38501
W20260811 03:10:14.112557 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 18931: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:14.112727 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 18931
2026-08-11T03:10:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:14 dist-test-slave-8tdl krb5kdc[19037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:14 dist-test-slave-8tdl krb5kdc[19037](info): set up 2 sockets
Aug 11 03:10:14 dist-test-slave-8tdl krb5kdc[19037](info): commencing operation
krb5kdc: starting...
W20260811 03:10:15.369997 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.006s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:15 dist-test-slave-8tdl krb5kdc[19037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:15.742377 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:16.117270 19051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:16.117674 19051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:16.117854 19051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:16.125357 19051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:16.125551 19051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:16.125681 19051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:16.125799 19051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:16.125891 19051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:16.136510 19051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:16.137964 19051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:16.139426 19051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:16.152369 19061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:16.152415 19058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:16.152460 19059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:16.153308 19051 server_base.cc:1034] running on GCE node
I20260811 03:10:16.154157 19051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:16.155848 19051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:16.157111 19051 hybrid_clock.cc:648] HybridClock initialized: now 1786417816157058 us; error 38 us; skew 500 ppm
I20260811 03:10:16.157521 19051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:16 dist-test-slave-8tdl krb5kdc[19037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:16.161072 19051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:16.162858 19051 webserver.cc:460] Webserver started at http://127.0.0.1:37865/ using document root <none> and password file <none>
I20260811 03:10:16.163623 19051 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:16.163782 19051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:16.164135 19051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:16.167476 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "5e0109c878904df4a900b290a2a7f3fb"
format_stamp: "Formatted at 2026-08-11 03:10:16 on dist-test-slave-8tdl"
server_key: "0d77b2b9e19f4b0426501900a22516a9"
server_key_iv: "ece3bc511389d5763afc5a41fbb2c269"
server_key_version: "encryptionkey@0"
I20260811 03:10:16.168329 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "5e0109c878904df4a900b290a2a7f3fb"
format_stamp: "Formatted at 2026-08-11 03:10:16 on dist-test-slave-8tdl"
server_key: "0d77b2b9e19f4b0426501900a22516a9"
server_key_iv: "ece3bc511389d5763afc5a41fbb2c269"
server_key_version: "encryptionkey@0"
I20260811 03:10:16.173810 19051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:10:16.177759 19069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:16.179383 19051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:10:16.179611 19051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "5e0109c878904df4a900b290a2a7f3fb"
format_stamp: "Formatted at 2026-08-11 03:10:16 on dist-test-slave-8tdl"
server_key: "0d77b2b9e19f4b0426501900a22516a9"
server_key_iv: "ece3bc511389d5763afc5a41fbb2c269"
server_key_version: "encryptionkey@0"
I20260811 03:10:16.179836 19051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:16.185986 19051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:16.191006 19051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:16.191339 19051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:16.211390 19051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46235
I20260811 03:10:16.211453 19126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46235 every 8 connection(s)
I20260811 03:10:16.213120 19051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:16.217831 19127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:16.220141 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 19051
I20260811 03:10:16.220367 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:16.220853 28084 external_mini_cluster.cc:1428] Setting key 275d9893cbb5612e0c7a332a880f3c83
I20260811 03:10:16.231248 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:16.223149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44677 (local address 127.0.0.1:46235)
0811 03:10:16.224082 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:16.224099 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:16.224155 (+ 56us) server_negotiation.cc:407] Connection header received
0811 03:10:16.226720 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:16.226756 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:16.227285 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:16.227797 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:16.228979 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:16.228993 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:16.229011 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:16.229034 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:16.229108 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:16.229145 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:16.229418 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:16.229556 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 03:10:16.229782 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":386,"threads_started":1}
I20260811 03:10:16.233414 19127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb: Bootstrap starting.
I20260811 03:10:16.236207 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:16.237705 19127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:16.239169 19127 log.cc:824] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:16.242825 19127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb: No bootstrap required, opened a new log
I20260811 03:10:16.245081 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:16.242013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56657 (local address 127.0.0.1:46235)
0811 03:10:16.242252 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:16.242262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:16.242773 (+ 511us) server_negotiation.cc:407] Connection header received
0811 03:10:16.243064 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:16.243068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:16.243118 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:16.243207 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:16.244323 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:16.244328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:16.244331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:16.244346 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:16.244387 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:16.244405 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:16.244502 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:16.244649 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 03:10:16.244699 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 03:10:16.251971 19127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0109c878904df4a900b290a2a7f3fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260811 03:10:16.252539 19127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:16.252699 19127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e0109c878904df4a900b290a2a7f3fb, State: Initialized, Role: FOLLOWER
I20260811 03:10:16.253454 19127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0109c878904df4a900b290a2a7f3fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260811 03:10:16.253759 19127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:16.253927 19127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:16.254117 19127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:16.256737 19127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0109c878904df4a900b290a2a7f3fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260811 03:10:16.257386 19127 leader_election.cc:302] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e0109c878904df4a900b290a2a7f3fb; no voters:
I20260811 03:10:16.258086 19127 leader_election.cc:288] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:16.258188 19138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:16.259057 19138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [term 1 LEADER]: Becoming Leader. State: Replica: 5e0109c878904df4a900b290a2a7f3fb, State: Running, Role: LEADER
I20260811 03:10:16.259680 19138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0109c878904df4a900b290a2a7f3fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } }
I20260811 03:10:16.260381 19127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:16.262566 19140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e0109c878904df4a900b290a2a7f3fb. Latest consensus state: current_term: 1 leader_uuid: "5e0109c878904df4a900b290a2a7f3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0109c878904df4a900b290a2a7f3fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } } }
I20260811 03:10:16.263077 19140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:16.262850 19139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e0109c878904df4a900b290a2a7f3fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0109c878904df4a900b290a2a7f3fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46235 } } }
I20260811 03:10:16.263242 19139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:16.266180 19143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:16.272053 19143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:16.282660 19143 catalog_manager.cc:1347] Generated new cluster ID: d6f8a17703f448139ccb08b2284bcff1
I20260811 03:10:16.282840 19143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:16.307871 19143 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:16.308379 19143 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:16.318356 19143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e0109c878904df4a900b290a2a7f3fb: Generated new TSK 0
I20260811 03:10:16.319370 19143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:16.333107 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19051
W20260811 03:10:16.334476 19122 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44677
W20260811 03:10:16.335481 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19051: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:16.335696 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 19051
2026-08-11T03:10:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:16 dist-test-slave-8tdl krb5kdc[19160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:16 dist-test-slave-8tdl krb5kdc[19160](info): set up 2 sockets
Aug 11 03:10:16 dist-test-slave-8tdl krb5kdc[19160](info): commencing operation
krb5kdc: starting...
W20260811 03:10:17.549698 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:17 dist-test-slave-8tdl krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:17.923815 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:18.299150 19173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:18.299468 19173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:18.299681 19173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:18.307324 19173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:18.307461 19173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:18.307581 19173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:18.307688 19173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:18.307768 19173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:18.318341 19173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:18.319849 19173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:18.321328 19173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:18.333899 19179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:18.333978 19182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:18.334002 19178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:18.335209 19173 server_base.cc:1034] running on GCE node
I20260811 03:10:18.336380 19173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:18.337972 19173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:18.339246 19173 hybrid_clock.cc:648] HybridClock initialized: now 1786417818339202 us; error 28 us; skew 500 ppm
I20260811 03:10:18.339686 19173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:18 dist-test-slave-8tdl krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:18.343644 19173 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:18.345460 19173 webserver.cc:460] Webserver started at http://127.0.0.1:36709/ using document root <none> and password file <none>
I20260811 03:10:18.346210 19173 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:18.346364 19173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:18.346771 19173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:18.350085 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "36d439e496c1411e9da5d15850940f5b"
format_stamp: "Formatted at 2026-08-11 03:10:18 on dist-test-slave-8tdl"
server_key: "1adbfacf6b5c0a098280fd1d1476943e"
server_key_iv: "33e9af6d5879b7280d595c0acfdcd16c"
server_key_version: "encryptionkey@0"
I20260811 03:10:18.351038 19173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "36d439e496c1411e9da5d15850940f5b"
format_stamp: "Formatted at 2026-08-11 03:10:18 on dist-test-slave-8tdl"
server_key: "1adbfacf6b5c0a098280fd1d1476943e"
server_key_iv: "33e9af6d5879b7280d595c0acfdcd16c"
server_key_version: "encryptionkey@0"
I20260811 03:10:18.356878 19173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:10:18.361150 19190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:18.362649 19173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:10:18.362983 19173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "36d439e496c1411e9da5d15850940f5b"
format_stamp: "Formatted at 2026-08-11 03:10:18 on dist-test-slave-8tdl"
server_key: "1adbfacf6b5c0a098280fd1d1476943e"
server_key_iv: "33e9af6d5879b7280d595c0acfdcd16c"
server_key_version: "encryptionkey@0"
I20260811 03:10:18.363255 19173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:18.388533 19173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:18.393208 19173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:18.393548 19173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:18.414361 19173 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34875
I20260811 03:10:18.414402 19245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34875 every 8 connection(s)
I20260811 03:10:18.415908 19173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:18.420104 19246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:18.421731 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 19173
I20260811 03:10:18.421931 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:18.422413 28084 external_mini_cluster.cc:1428] Setting key 30f1d0e541762023a8aad7373e5cbe14
I20260811 03:10:18.435935 19246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b: Bootstrap starting.
Aug 11 03:10:18 dist-test-slave-8tdl krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 03:10:18.440047 19246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:18.441243 19246 log.cc:824] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:18.444475 19246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b: No bootstrap required, opened a new log
I20260811 03:10:18.448045 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:18.424681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:51845 (local address 127.0.0.1:34875)
0811 03:10:18.425589 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:18.425604 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:18.425647 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:10:18.427378 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:18.427404 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:18.427868 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:18.428247 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:18.430182 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:18.431698 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:18.433626 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:18.434353 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:18.434486 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:18.439703 (+ 5217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:18.439712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:18.439728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:18.439776 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:18.443113 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:18.444070 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:18.444077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:18.444083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:18.444268 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:18.444820 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:18.444823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:18.444825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:18.445392 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:18.445767 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:18.446363 (+ 596us) server_negotiation.cc:299] Negotiation successful
0811 03:10:18.446842 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":455,"threads_started":1}
I20260811 03:10:18.452432 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:18.453398 19246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d439e496c1411e9da5d15850940f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260811 03:10:18.453792 19246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:18.453913 19246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d439e496c1411e9da5d15850940f5b, State: Initialized, Role: FOLLOWER
I20260811 03:10:18.454481 19246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d439e496c1411e9da5d15850940f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260811 03:10:18.454752 19246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:18.454938 19246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:18.455118 19246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:18.457769 19246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d439e496c1411e9da5d15850940f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260811 03:10:18.458500 19246 leader_election.cc:302] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d439e496c1411e9da5d15850940f5b; no voters:
I20260811 03:10:18.459352 19246 leader_election.cc:288] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:18.459482 19256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:18.460333 19256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [term 1 LEADER]: Becoming Leader. State: Replica: 36d439e496c1411e9da5d15850940f5b, State: Running, Role: LEADER
I20260811 03:10:18.460865 19256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d439e496c1411e9da5d15850940f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } }
I20260811 03:10:18.463039 19246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:18.464006 19259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d439e496c1411e9da5d15850940f5b. Latest consensus state: current_term: 1 leader_uuid: "36d439e496c1411e9da5d15850940f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d439e496c1411e9da5d15850940f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } } }
I20260811 03:10:18.464432 19259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:18.469332 19257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36d439e496c1411e9da5d15850940f5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d439e496c1411e9da5d15850940f5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34875 } } }
I20260811 03:10:18.469712 19257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:18.476810 19262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:18.481199 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:18.458048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:55641 (local address 127.0.0.1:34875)
0811 03:10:18.458270 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:18.458276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:18.458649 (+ 373us) server_negotiation.cc:407] Connection header received
0811 03:10:18.458956 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:18.458962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:18.459066 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:18.459212 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:18.461433 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:18.462716 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:18.465012 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:18.466010 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:18.466147 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:18.471275 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:18.471287 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:18.471295 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:18.471374 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:18.476279 (+ 4905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:18.477480 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:18.477487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:18.477490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:18.477672 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:18.478296 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:18.478301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:18.478304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:18.478851 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:18.479192 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:18.480545 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0811 03:10:18.480761 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260811 03:10:18.482837 19262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:18.493933 19262 catalog_manager.cc:1347] Generated new cluster ID: 9d7639a22ff44e81aa7eb377bec17a77
I20260811 03:10:18.494091 19262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:18.511500 19262 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:18.513137 19262 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:18.525084 19262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36d439e496c1411e9da5d15850940f5b: Generated new TSK 0
I20260811 03:10:18.526067 19262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:18.538964 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19173
I20260811 03:10:18.540726 19241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:18.634604 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 19173
W20260811 03:10:18.657678 19158 connection.cc:535] client connection to 127.0.0.1:34875 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34875): Connection reset by peer (error 104)
2026-08-11T03:10:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:18 dist-test-slave-8tdl krb5kdc[19283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:18 dist-test-slave-8tdl krb5kdc[19283](info): set up 2 sockets
Aug 11 03:10:18 dist-test-slave-8tdl krb5kdc[19283](info): commencing operation
krb5kdc: starting...
W20260811 03:10:19.888195 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.001s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:20 dist-test-slave-8tdl krb5kdc[19283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:20.270879 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:20.644711 19297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:20.645104 19297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:20.645288 19297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:20.652791 19297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:20.652998 19297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:20.653143 19297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:20.653260 19297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:20.653352 19297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:20.664073 19297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:20.665592 19297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:20.667038 19297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:20.680106 19305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:20.680131 19303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:20.680675 19302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:20.681308 19297 server_base.cc:1034] running on GCE node
I20260811 03:10:20.682247 19297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:20.683941 19297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:20.685205 19297 hybrid_clock.cc:648] HybridClock initialized: now 1786417820685158 us; error 67 us; skew 500 ppm
I20260811 03:10:20.685597 19297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:20 dist-test-slave-8tdl krb5kdc[19283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:20.695281 19297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:20.697288 19297 webserver.cc:460] Webserver started at http://127.0.0.1:34261/ using document root <none> and password file <none>
I20260811 03:10:20.698092 19297 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:20.698261 19297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:20.698627 19297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:20.702064 19297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "7114fbea2b374703962e022825bd72df"
format_stamp: "Formatted at 2026-08-11 03:10:20 on dist-test-slave-8tdl"
server_key: "309af3a52a8691806bec5d3e19639cec"
server_key_iv: "4ffbcfe7c059e6bac487f0f02a66624f"
server_key_version: "encryptionkey@0"
I20260811 03:10:20.703050 19297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "7114fbea2b374703962e022825bd72df"
format_stamp: "Formatted at 2026-08-11 03:10:20 on dist-test-slave-8tdl"
server_key: "309af3a52a8691806bec5d3e19639cec"
server_key_iv: "4ffbcfe7c059e6bac487f0f02a66624f"
server_key_version: "encryptionkey@0"
I20260811 03:10:20.709079 19297 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260811 03:10:20.713145 19312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:20.714848 19297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:10:20.715075 19297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "7114fbea2b374703962e022825bd72df"
format_stamp: "Formatted at 2026-08-11 03:10:20 on dist-test-slave-8tdl"
server_key: "309af3a52a8691806bec5d3e19639cec"
server_key_iv: "4ffbcfe7c059e6bac487f0f02a66624f"
server_key_version: "encryptionkey@0"
I20260811 03:10:20.715328 19297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:20.746496 19297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:20.751518 19297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:20.751864 19297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:20.772854 19297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34429
I20260811 03:10:20.772898 19368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34429 every 8 connection(s)
I20260811 03:10:20.774495 19297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:20.779037 19369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:20.779407 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 19297
I20260811 03:10:20.779609 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:20.780155 28084 external_mini_cluster.cc:1428] Setting key 1ab0d98f00acbbaa41c677143349b6c6
I20260811 03:10:20.795135 19369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df: Bootstrap starting.
I20260811 03:10:20.795157 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:20.782180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:53393 (local address 127.0.0.1:34429)
0811 03:10:20.783102 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:20.783122 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:20.783166 (+ 44us) server_negotiation.cc:407] Connection header received
0811 03:10:20.784763 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:20.784787 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:20.785226 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:20.785632 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:20.786284 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:20.787915 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:20.789820 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:20.790588 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:20.790804 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:20.792239 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:20.792252 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:20.792269 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:20.793003 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:20.793089 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:20.793122 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:20.793261 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:20.793642 (+ 381us) server_negotiation.cc:299] Negotiation successful
0811 03:10:20.793954 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":460,"threads_started":1}
I20260811 03:10:20.799155 19369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:20.800045 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:20.800482 19369 log.cc:824] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:20.804301 19369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df: No bootstrap required, opened a new log
I20260811 03:10:20.814083 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:20.805699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:46361 (local address 127.0.0.1:34429)
0811 03:10:20.805936 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:20.805942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:20.807391 (+ 1449us) server_negotiation.cc:407] Connection header received
0811 03:10:20.807559 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:20.807565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:20.807683 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:20.807815 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:20.808394 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:20.809601 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:20.811124 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:20.811855 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:20.812002 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:20.813267 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:20.813275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:20.813279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:20.813307 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:20.813377 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:20.813398 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:20.813532 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:20.813686 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 03:10:20.813753 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 03:10:20.814265 19369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7114fbea2b374703962e022825bd72df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } }
I20260811 03:10:20.814787 19369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:20.814941 19369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7114fbea2b374703962e022825bd72df, State: Initialized, Role: FOLLOWER
I20260811 03:10:20.815693 19369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7114fbea2b374703962e022825bd72df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } }
I20260811 03:10:20.816005 19369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:20.816169 19369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:20.816355 19369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:20.818970 19369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7114fbea2b374703962e022825bd72df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } }
I20260811 03:10:20.819540 19369 leader_election.cc:302] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7114fbea2b374703962e022825bd72df; no voters:
I20260811 03:10:20.820338 19369 leader_election.cc:288] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:20.820648 19379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:20.821398 19379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [term 1 LEADER]: Becoming Leader. State: Replica: 7114fbea2b374703962e022825bd72df, State: Running, Role: LEADER
I20260811 03:10:20.822075 19379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7114fbea2b374703962e022825bd72df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } }
I20260811 03:10:20.822731 19369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:20.824957 19380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7114fbea2b374703962e022825bd72df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7114fbea2b374703962e022825bd72df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } } }
I20260811 03:10:20.825346 19380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:20.826321 19384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:20.826251 19381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7114fbea2b374703962e022825bd72df. Latest consensus state: current_term: 1 leader_uuid: "7114fbea2b374703962e022825bd72df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7114fbea2b374703962e022825bd72df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34429 } } }
I20260811 03:10:20.827111 19381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:20.833910 19384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:20.843544 19384 catalog_manager.cc:1347] Generated new cluster ID: b05cf40b634941d38075a26543c188bd
I20260811 03:10:20.843689 19384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:20.867431 19384 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:20.869168 19384 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:20.885756 19384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7114fbea2b374703962e022825bd72df: Generated new TSK 0
I20260811 03:10:20.886830 19384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:20.901072 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19297
W20260811 03:10:20.902552 19364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:53393
W20260811 03:10:20.903582 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19297: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:20.903756 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 19297
W20260811 03:10:20.925892 19281 connection.cc:535] client connection to 127.0.0.1:34429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34429): Connection reset by peer (error 104)
2026-08-11T03:10:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2268 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:21 dist-test-slave-8tdl krb5kdc[19400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:21 dist-test-slave-8tdl krb5kdc[19400](info): set up 2 sockets
Aug 11 03:10:21 dist-test-slave-8tdl krb5kdc[19400](info): commencing operation
krb5kdc: starting...
W20260811 03:10:22.139389 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:22 dist-test-slave-8tdl krb5kdc[19400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:22.515271 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:22.884788 19416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:22.885226 19416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:22.885401 19416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:22.893729 19416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:22.893944 19416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:22.894055 19416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:22.894167 19416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:22.894245 19416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:22.904958 19416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:22.906438 19416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:22.907895 19416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:22.920217 19422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:22.920292 19424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:22.920315 19421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:22.921454 19416 server_base.cc:1034] running on GCE node
I20260811 03:10:22.922353 19416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:22.923923 19416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:22.925200 19416 hybrid_clock.cc:648] HybridClock initialized: now 1786417822925138 us; error 66 us; skew 500 ppm
I20260811 03:10:22.925648 19416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:22 dist-test-slave-8tdl krb5kdc[19400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:22.929138 19416 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:22.930820 19416 webserver.cc:460] Webserver started at http://127.0.0.1:43469/ using document root <none> and password file <none>
I20260811 03:10:22.931519 19416 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:22.931631 19416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:22.931903 19416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:22.935377 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "fcb4aba573844a67b090b14d0aac00b1"
format_stamp: "Formatted at 2026-08-11 03:10:22 on dist-test-slave-8tdl"
server_key: "fe6140d5fd31868ce0bdbd5d38d73a2b"
server_key_iv: "299c1d240d9c65bb32014a66cf411540"
server_key_version: "encryptionkey@0"
I20260811 03:10:22.936237 19416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "fcb4aba573844a67b090b14d0aac00b1"
format_stamp: "Formatted at 2026-08-11 03:10:22 on dist-test-slave-8tdl"
server_key: "fe6140d5fd31868ce0bdbd5d38d73a2b"
server_key_iv: "299c1d240d9c65bb32014a66cf411540"
server_key_version: "encryptionkey@0"
I20260811 03:10:22.942097 19416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 03:10:22.946228 19433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:22.947810 19416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:10:22.948096 19416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "fcb4aba573844a67b090b14d0aac00b1"
format_stamp: "Formatted at 2026-08-11 03:10:22 on dist-test-slave-8tdl"
server_key: "fe6140d5fd31868ce0bdbd5d38d73a2b"
server_key_iv: "299c1d240d9c65bb32014a66cf411540"
server_key_version: "encryptionkey@0"
I20260811 03:10:22.948307 19416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:22.954833 19416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:22.959766 19416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:22.960109 19416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:22.979131 19416 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44519
I20260811 03:10:22.979182 19489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44519 every 8 connection(s)
I20260811 03:10:22.980724 19416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:22.983439 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 19416
I20260811 03:10:22.983631 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:22.984186 28084 external_mini_cluster.cc:1428] Setting key d44b6affd71baca6ca97977712fd1001
I20260811 03:10:22.985169 19491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:22.993216 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:22.986078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:46789 (local address 127.0.0.1:44519)
0811 03:10:22.986964 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:22.986979 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:22.987043 (+ 64us) server_negotiation.cc:407] Connection header received
0811 03:10:22.988932 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:22.988962 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:22.989478 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:22.989957 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:22.991057 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:22.991068 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:22.991092 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:22.991119 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:22.991206 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:22.991242 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:22.991478 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:22.991620 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 03:10:22.991840 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":334,"threads_started":1}
I20260811 03:10:22.997784 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:23.001672 19491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1: Bootstrap starting.
I20260811 03:10:23.006398 19491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:23.006819 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:23.003576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:53775 (local address 127.0.0.1:44519)
0811 03:10:23.003840 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:23.003850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:23.004280 (+ 430us) server_negotiation.cc:407] Connection header received
0811 03:10:23.004478 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:23.004481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:23.004534 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:23.004624 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:23.005705 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:23.005710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:23.005713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:23.005731 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:23.005776 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:23.005793 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:23.005959 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:23.006179 (+ 220us) server_negotiation.cc:299] Negotiation successful
0811 03:10:23.006236 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 03:10:23.008121 19491 log.cc:824] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:23.011847 19491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1: No bootstrap required, opened a new log
I20260811 03:10:23.022203 19491 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb4aba573844a67b090b14d0aac00b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44519 } }
I20260811 03:10:23.022804 19491 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:23.022910 19491 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb4aba573844a67b090b14d0aac00b1, State: Initialized, Role: FOLLOWER
I20260811 03:10:23.023644 19491 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb4aba573844a67b090b14d0aac00b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44519 } }
I20260811 03:10:23.023958 19491 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:23.024139 19491 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:23.024359 19491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:23.027081 19491 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb4aba573844a67b090b14d0aac00b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44519 } }
I20260811 03:10:23.027595 19491 leader_election.cc:302] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcb4aba573844a67b090b14d0aac00b1; no voters:
I20260811 03:10:23.028211 19491 leader_election.cc:288] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:23.028393 19501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:23.029249 19501 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [term 1 LEADER]: Becoming Leader. State: Replica: fcb4aba573844a67b090b14d0aac00b1, State: Running, Role: LEADER
I20260811 03:10:23.029850 19501 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb4aba573844a67b090b14d0aac00b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44519 } }
I20260811 03:10:23.030490 19491 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:23.032980 19503 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb4aba573844a67b090b14d0aac00b1. Latest consensus state: current_term: 1 leader_uuid: "fcb4aba573844a67b090b14d0aac00b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb4aba573844a67b090b14d0aac00b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44519 } } }
I20260811 03:10:23.033555 19503 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:23.033131 19502 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcb4aba573844a67b090b14d0aac00b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb4aba573844a67b090b14d0aac00b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44519 } } }
I20260811 03:10:23.034021 19502 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:23.035363 19510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:23.041501 19510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:23.050565 19510 catalog_manager.cc:1347] Generated new cluster ID: ef80874d6da04c2397cfb9c8a6e7d887
I20260811 03:10:23.050719 19510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:23.088893 19510 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:23.089437 19510 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:23.097149 19510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcb4aba573844a67b090b14d0aac00b1: Generated new TSK 0
I20260811 03:10:23.098044 19510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:23.103580 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19416
W20260811 03:10:23.105405 19485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:46789
W20260811 03:10:23.106213 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19416: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:23.106397 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 19416
2026-08-11T03:10:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:23 dist-test-slave-8tdl krb5kdc[19523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:23 dist-test-slave-8tdl krb5kdc[19523](info): set up 2 sockets
Aug 11 03:10:23 dist-test-slave-8tdl krb5kdc[19523](info): commencing operation
krb5kdc: starting...
W20260811 03:10:24.388829 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:24 dist-test-slave-8tdl krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:24.762538 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:25.142239 19536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:25.142571 19536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:25.142669 19536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:25.150475 19536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:25.150601 19536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:25.150650 19536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:25.150781 19536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:25.150833 19536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:25.161401 19536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:25.162868 19536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:25.164256 19536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:25.177181 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:25.177213 19545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:25.177165 19543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:25.177838 19536 server_base.cc:1034] running on GCE node
I20260811 03:10:25.178864 19536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:25.180392 19536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:25.181596 19536 hybrid_clock.cc:648] HybridClock initialized: now 1786417825181532 us; error 53 us; skew 500 ppm
I20260811 03:10:25.181932 19536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:25 dist-test-slave-8tdl krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:25.185338 19536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:25.187062 19536 webserver.cc:460] Webserver started at http://127.0.0.1:36363/ using document root <none> and password file <none>
I20260811 03:10:25.187764 19536 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:25.187871 19536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:25.188166 19536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:25.191401 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "a668b8988c2d4b12af3bc7b553f2435c"
format_stamp: "Formatted at 2026-08-11 03:10:25 on dist-test-slave-8tdl"
server_key: "d078387f78379033ee7c8d614edb7691"
server_key_iv: "2b4b11bf2d6ebf281074be5a34d0a7ac"
server_key_version: "encryptionkey@0"
I20260811 03:10:25.192200 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "a668b8988c2d4b12af3bc7b553f2435c"
format_stamp: "Formatted at 2026-08-11 03:10:25 on dist-test-slave-8tdl"
server_key: "d078387f78379033ee7c8d614edb7691"
server_key_iv: "2b4b11bf2d6ebf281074be5a34d0a7ac"
server_key_version: "encryptionkey@0"
I20260811 03:10:25.197898 19536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 03:10:25.202080 19552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:25.203800 19536 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260811 03:10:25.203986 19536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "a668b8988c2d4b12af3bc7b553f2435c"
format_stamp: "Formatted at 2026-08-11 03:10:25 on dist-test-slave-8tdl"
server_key: "d078387f78379033ee7c8d614edb7691"
server_key_iv: "2b4b11bf2d6ebf281074be5a34d0a7ac"
server_key_version: "encryptionkey@0"
I20260811 03:10:25.204244 19536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:25.229763 19536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:25.234874 19536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:25.235157 19536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:25.256259 19536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40609
I20260811 03:10:25.256279 19606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40609 every 8 connection(s)
I20260811 03:10:25.257840 19536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:25.261111 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 19536
I20260811 03:10:25.261315 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:25.261811 28084 external_mini_cluster.cc:1428] Setting key fa521255521dba19c456a74b64f15cbb
I20260811 03:10:25.262245 19608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 03:10:25 dist-test-slave-8tdl krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 03:10:25.278946 19608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c: Bootstrap starting.
I20260811 03:10:25.283051 19608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:25.284214 19608 log.cc:824] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:25.287602 19608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c: No bootstrap required, opened a new log
I20260811 03:10:25.287635 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:25.263738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:54275 (local address 127.0.0.1:40609)
0811 03:10:25.264565 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:25.264578 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:25.264621 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:10:25.266296 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:25.266319 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:25.266821 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:25.267217 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:25.269189 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:25.270618 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:25.272523 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:25.273327 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:25.273463 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:25.278936 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:25.278946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:25.278964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:25.279016 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:25.282654 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:25.283818 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:25.283826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:25.283832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:25.284030 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:25.284593 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:25.284600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:25.284609 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:25.285212 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:25.285587 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:25.286173 (+ 586us) server_negotiation.cc:299] Negotiation successful
0811 03:10:25.286504 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":381,"threads_started":1}
I20260811 03:10:25.292397 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:25.296813 19608 raft_consensus.cc:348] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a668b8988c2d4b12af3bc7b553f2435c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40609 } }
I20260811 03:10:25.297576 19608 raft_consensus.cc:374] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:25.297734 19608 raft_consensus.cc:729] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a668b8988c2d4b12af3bc7b553f2435c, State: Initialized, Role: FOLLOWER
I20260811 03:10:25.298620 19608 consensus_queue.cc:260] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a668b8988c2d4b12af3bc7b553f2435c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40609 } }
I20260811 03:10:25.298938 19608 raft_consensus.cc:388] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:25.299104 19608 raft_consensus.cc:482] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:25.299305 19608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:25.301744 19608 raft_consensus.cc:504] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a668b8988c2d4b12af3bc7b553f2435c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40609 } }
I20260811 03:10:25.302292 19608 leader_election.cc:302] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a668b8988c2d4b12af3bc7b553f2435c; no voters:
I20260811 03:10:25.302997 19608 leader_election.cc:288] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:25.303159 19618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:25.305639 19608 sys_catalog.cc:564] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:25.307027 19618 raft_consensus.cc:686] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [term 1 LEADER]: Becoming Leader. State: Replica: a668b8988c2d4b12af3bc7b553f2435c, State: Running, Role: LEADER
I20260811 03:10:25.307721 19618 consensus_queue.cc:237] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a668b8988c2d4b12af3bc7b553f2435c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40609 } }
I20260811 03:10:25.315393 19620 sys_catalog.cc:455] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a668b8988c2d4b12af3bc7b553f2435c. Latest consensus state: current_term: 1 leader_uuid: "a668b8988c2d4b12af3bc7b553f2435c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a668b8988c2d4b12af3bc7b553f2435c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40609 } } }
I20260811 03:10:25.315841 19620 sys_catalog.cc:458] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:25.319929 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:25.297286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:34201 (local address 127.0.0.1:40609)
0811 03:10:25.297456 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:25.297461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:25.298192 (+ 731us) server_negotiation.cc:407] Connection header received
0811 03:10:25.298405 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:25.298414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:25.298483 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:25.298594 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:25.300318 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:25.301140 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:25.302589 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:25.303568 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:25.307251 (+ 3683us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:25.307635 (+ 384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:25.307644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:25.307652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:25.307696 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:25.313006 (+ 5310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:25.314078 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:25.314095 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:25.314102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:25.314321 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:25.315153 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:25.315160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:25.315165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:25.315642 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:25.317574 (+ 1932us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:25.318961 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0811 03:10:25.319309 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 03:10:25.316495 19619 sys_catalog.cc:455] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a668b8988c2d4b12af3bc7b553f2435c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a668b8988c2d4b12af3bc7b553f2435c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40609 } } }
I20260811 03:10:25.320333 19619 sys_catalog.cc:458] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:25.322450 19624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:25.329537 19624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:25.340297 19624 catalog_manager.cc:1347] Generated new cluster ID: b66115660e0e43eaa51fea9c9dc60cdc
I20260811 03:10:25.340443 19624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:25.364305 19624 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:25.365885 19624 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:25.375711 19624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a668b8988c2d4b12af3bc7b553f2435c: Generated new TSK 0
I20260811 03:10:25.376818 19624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:25.389966 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19536
I20260811 03:10:25.392230 19602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:25.488777 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 19536
W20260811 03:10:25.511340 19521 connection.cc:535] client connection to 127.0.0.1:40609 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40609): Connection reset by peer (error 104)
2026-08-11T03:10:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:25 dist-test-slave-8tdl krb5kdc[19642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:25 dist-test-slave-8tdl krb5kdc[19642](info): set up 2 sockets
Aug 11 03:10:25 dist-test-slave-8tdl krb5kdc[19642](info): commencing operation
krb5kdc: starting...
W20260811 03:10:26.707093 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:26 dist-test-slave-8tdl krb5kdc[19642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:27.090978 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:27.466491 19657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:27.466925 19657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:27.467142 19657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:27.475021 19657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:27.475195 19657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:27.475287 19657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:27.475383 19657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:27.475450 19657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:27.486111 19657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:27.487672 19657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:27.489084 19657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:27.502918 19667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:27.503206 19663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:27.503121 19664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:27.504571 19657 server_base.cc:1034] running on GCE node
I20260811 03:10:27.505497 19657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:27.507222 19657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:27.508448 19657 hybrid_clock.cc:648] HybridClock initialized: now 1786417827508393 us; error 42 us; skew 500 ppm
I20260811 03:10:27.508849 19657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:27 dist-test-slave-8tdl krb5kdc[19642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:27.516206 19657 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:27.518002 19657 webserver.cc:460] Webserver started at http://127.0.0.1:34619/ using document root <none> and password file <none>
I20260811 03:10:27.518914 19657 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:27.519100 19657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:27.519462 19657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:27.523054 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "87b9bc481a644d82af355c821701f7f1"
format_stamp: "Formatted at 2026-08-11 03:10:27 on dist-test-slave-8tdl"
server_key: "9aee3bd1ceedd3f3a55f78e6a4866e76"
server_key_iv: "11d9882a48fe67a538514b06fb6530c8"
server_key_version: "encryptionkey@0"
I20260811 03:10:27.523962 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "87b9bc481a644d82af355c821701f7f1"
format_stamp: "Formatted at 2026-08-11 03:10:27 on dist-test-slave-8tdl"
server_key: "9aee3bd1ceedd3f3a55f78e6a4866e76"
server_key_iv: "11d9882a48fe67a538514b06fb6530c8"
server_key_version: "encryptionkey@0"
I20260811 03:10:27.530025 19657 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260811 03:10:27.534152 19674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:27.535815 19657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 03:10:27.536043 19657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "87b9bc481a644d82af355c821701f7f1"
format_stamp: "Formatted at 2026-08-11 03:10:27 on dist-test-slave-8tdl"
server_key: "9aee3bd1ceedd3f3a55f78e6a4866e76"
server_key_iv: "11d9882a48fe67a538514b06fb6530c8"
server_key_version: "encryptionkey@0"
I20260811 03:10:27.536294 19657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:27.552102 19657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:27.557422 19657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:27.557777 19657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:27.578644 19657 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44833
I20260811 03:10:27.578769 19728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44833 every 8 connection(s)
I20260811 03:10:27.580250 19657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:27.584263 19729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:27.588126 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 19657
I20260811 03:10:27.588316 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:27.588889 28084 external_mini_cluster.cc:1428] Setting key b0c411fbe4c7f9d98f7552cc8eac445c
I20260811 03:10:27.599330 19729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1: Bootstrap starting.
I20260811 03:10:27.602862 19729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:27.604116 19729 log.cc:824] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:27.608006 19729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1: No bootstrap required, opened a new log
I20260811 03:10:27.611245 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:27.597136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:60943 (local address 127.0.0.1:44833)
0811 03:10:27.598036 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:27.598050 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:27.598099 (+ 49us) server_negotiation.cc:407] Connection header received
0811 03:10:27.600289 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:27.600316 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:27.600763 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:27.601228 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:27.601863 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:27.603546 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:27.605708 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:27.606805 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:27.607017 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:27.608376 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:27.608384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:27.608430 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:27.609078 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:27.609168 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:27.609197 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:27.609389 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:27.609762 (+ 373us) server_negotiation.cc:299] Negotiation successful
0811 03:10:27.609970 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":399,"threads_started":1}
I20260811 03:10:27.616227 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:27.617550 19729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9bc481a644d82af355c821701f7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260811 03:10:27.618018 19729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:27.618137 19729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b9bc481a644d82af355c821701f7f1, State: Initialized, Role: FOLLOWER
I20260811 03:10:27.618892 19729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9bc481a644d82af355c821701f7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260811 03:10:27.619190 19729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:27.619340 19729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:27.619560 19729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:27.622498 19729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9bc481a644d82af355c821701f7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260811 03:10:27.623207 19729 leader_election.cc:302] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b9bc481a644d82af355c821701f7f1; no voters:
I20260811 03:10:27.623937 19729 leader_election.cc:288] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:27.624089 19739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:27.626221 19729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:27.626796 19739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [term 1 LEADER]: Becoming Leader. State: Replica: 87b9bc481a644d82af355c821701f7f1, State: Running, Role: LEADER
I20260811 03:10:27.627516 19739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9bc481a644d82af355c821701f7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260811 03:10:27.630158 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:27.621240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:54953 (local address 127.0.0.1:44833)
0811 03:10:27.621459 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:27.621467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:27.621792 (+ 325us) server_negotiation.cc:407] Connection header received
0811 03:10:27.621943 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:27.621948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:27.622032 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:27.622148 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:27.622654 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:27.623868 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:27.626475 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:27.627274 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:27.627431 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:27.628859 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:27.628869 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:27.628874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:27.628897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:27.628991 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:27.629023 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:27.629150 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:27.629729 (+ 579us) server_negotiation.cc:299] Negotiation successful
0811 03:10:27.629800 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 03:10:27.631085 19742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b9bc481a644d82af355c821701f7f1. Latest consensus state: current_term: 1 leader_uuid: "87b9bc481a644d82af355c821701f7f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9bc481a644d82af355c821701f7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } } }
I20260811 03:10:27.631333 19740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b9bc481a644d82af355c821701f7f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b9bc481a644d82af355c821701f7f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } } }
I20260811 03:10:27.631577 19742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:27.631646 19740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:27.636750 19751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:27.642335 19751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:27.652786 19751 catalog_manager.cc:1347] Generated new cluster ID: 6463f7eb279c40d2b9b57d60204eeec9
I20260811 03:10:27.652974 19751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:27.668455 19751 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:27.670078 19751 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:27.682535 19751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b9bc481a644d82af355c821701f7f1: Generated new TSK 0
I20260811 03:10:27.683688 19751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:27.690714 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19657
W20260811 03:10:27.692435 19724 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:60943
W20260811 03:10:27.693390 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19657: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:27.693567 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 19657
W20260811 03:10:27.713164 19640 connection.cc:535] client connection to 127.0.0.1:44833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44833): Connection reset by peer (error 104)
2026-08-11T03:10:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:27 dist-test-slave-8tdl krb5kdc[19763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:27 dist-test-slave-8tdl krb5kdc[19763](info): set up 2 sockets
Aug 11 03:10:27 dist-test-slave-8tdl krb5kdc[19763](info): commencing operation
krb5kdc: starting...
W20260811 03:10:28.937413 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:29 dist-test-slave-8tdl krb5kdc[19763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 03:10:29.319547 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:29.711162 19777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:29.711501 19777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:29.711668 19777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:29.719310 19777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:29.719439 19777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:29.719548 19777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:29.719656 19777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:29.719738 19777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:29.733860 19777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:29.735607 19777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:29.737151 19777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:29.751272 19787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:29.751312 19783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:29.751312 19784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:29.752688 19777 server_base.cc:1034] running on GCE node
I20260811 03:10:29.753736 19777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:29.755522 19777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:29.756750 19777 hybrid_clock.cc:648] HybridClock initialized: now 1786417829756709 us; error 34 us; skew 500 ppm
I20260811 03:10:29.757165 19777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:29 dist-test-slave-8tdl krb5kdc[19763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:29.760854 19777 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 03:10:29.762916 19777 webserver.cc:460] Webserver started at http://127.0.0.1:42813/ using document root <none> and password file <none>
I20260811 03:10:29.763744 19777 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:29.763928 19777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:29.764292 19777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:29.767730 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "e3cec753f6eb4019af78f1ea7cf65922"
format_stamp: "Formatted at 2026-08-11 03:10:29 on dist-test-slave-8tdl"
server_key: "e94ebb8f636bde1672092c9a69a499b7"
server_key_iv: "890e2ac5079ea2e99cbcb1416d873da1"
server_key_version: "encryptionkey@0"
I20260811 03:10:29.768646 19777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "e3cec753f6eb4019af78f1ea7cf65922"
format_stamp: "Formatted at 2026-08-11 03:10:29 on dist-test-slave-8tdl"
server_key: "e94ebb8f636bde1672092c9a69a499b7"
server_key_iv: "890e2ac5079ea2e99cbcb1416d873da1"
server_key_version: "encryptionkey@0"
I20260811 03:10:29.774951 19777 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 03:10:29.779395 19794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:29.780987 19777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:10:29.781212 19777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "e3cec753f6eb4019af78f1ea7cf65922"
format_stamp: "Formatted at 2026-08-11 03:10:29 on dist-test-slave-8tdl"
server_key: "e94ebb8f636bde1672092c9a69a499b7"
server_key_iv: "890e2ac5079ea2e99cbcb1416d873da1"
server_key_version: "encryptionkey@0"
I20260811 03:10:29.781457 19777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:29.788724 19777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:29.793984 19777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:29.794332 19777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:29.815340 19777 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46523
I20260811 03:10:29.815438 19852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46523 every 8 connection(s)
I20260811 03:10:29.816886 19777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:29.818825 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 19777
I20260811 03:10:29.819010 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:29.819491 28084 external_mini_cluster.cc:1428] Setting key c36491a54941f43c582306b0438eb39d
I20260811 03:10:29.821439 19853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:29.828498 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:29.821505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:55475 (local address 127.0.0.1:46523)
0811 03:10:29.822369 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:29.822387 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:29.822444 (+ 57us) server_negotiation.cc:407] Connection header received
0811 03:10:29.824697 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:29.824719 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:29.825092 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:29.825460 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:29.826632 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:29.826639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:29.826651 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:29.826718 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:29.826806 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:29.826832 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:29.827078 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:29.827184 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 03:10:29.827436 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":320,"threads_started":1}
I20260811 03:10:29.833063 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:29.836968 19853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922: Bootstrap starting.
I20260811 03:10:29.841379 19853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:29.842099 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:29.838995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.51:39295 (local address 127.0.0.1:46523)
0811 03:10:29.839246 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:29.839253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:29.839692 (+ 439us) server_negotiation.cc:407] Connection header received
0811 03:10:29.839846 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:29.839853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:29.839925 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:29.840060 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:29.841166 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:29.841175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:29.841180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:29.841202 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:29.841271 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:29.841294 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:29.841437 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:29.841660 (+ 223us) server_negotiation.cc:299] Negotiation successful
0811 03:10:29.841735 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 03:10:29.842986 19853 log.cc:824] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:29.846501 19853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922: No bootstrap required, opened a new log
I20260811 03:10:29.855741 19853 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cec753f6eb4019af78f1ea7cf65922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260811 03:10:29.856184 19853 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:29.856264 19853 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cec753f6eb4019af78f1ea7cf65922, State: Initialized, Role: FOLLOWER
I20260811 03:10:29.856818 19853 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cec753f6eb4019af78f1ea7cf65922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260811 03:10:29.857031 19853 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:29.857141 19853 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:29.857273 19853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:29.860191 19853 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cec753f6eb4019af78f1ea7cf65922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260811 03:10:29.860759 19853 leader_election.cc:302] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3cec753f6eb4019af78f1ea7cf65922; no voters:
I20260811 03:10:29.861457 19853 leader_election.cc:288] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:29.861642 19863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:29.862371 19863 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [term 1 LEADER]: Becoming Leader. State: Replica: e3cec753f6eb4019af78f1ea7cf65922, State: Running, Role: LEADER
I20260811 03:10:29.863045 19863 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cec753f6eb4019af78f1ea7cf65922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260811 03:10:29.863853 19853 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:29.866446 19865 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3cec753f6eb4019af78f1ea7cf65922. Latest consensus state: current_term: 1 leader_uuid: "e3cec753f6eb4019af78f1ea7cf65922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cec753f6eb4019af78f1ea7cf65922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } } }
I20260811 03:10:29.866858 19864 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3cec753f6eb4019af78f1ea7cf65922" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cec753f6eb4019af78f1ea7cf65922" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } } }
I20260811 03:10:29.867123 19865 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:29.867165 19864 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:29.869391 19873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:29.875149 19873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:29.885265 19873 catalog_manager.cc:1347] Generated new cluster ID: 920b340714c9478599ced34c97e2fc5f
I20260811 03:10:29.885428 19873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:29.908478 19873 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:29.909004 19873 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:29.919301 19873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3cec753f6eb4019af78f1ea7cf65922: Generated new TSK 0
I20260811 03:10:29.920337 19873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:29.928778 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19777
W20260811 03:10:29.930157 19848 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.51:55475
W20260811 03:10:29.931037 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19777: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:29.931201 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 19777
2026-08-11T03:10:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:30 dist-test-slave-8tdl krb5kdc[19886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:30 dist-test-slave-8tdl krb5kdc[19886](info): set up 2 sockets
Aug 11 03:10:30 dist-test-slave-8tdl krb5kdc[19886](info): commencing operation
krb5kdc: starting...
W20260811 03:10:31.159063 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:31 dist-test-slave-8tdl krb5kdc[19886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:10:31.543291 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37327
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:37327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:31.919049 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:31.919450 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:31.919625 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:31.927277 19899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:31.927480 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:31.927600 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:31.927716 19899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:31.927807 19899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:31.938472 19899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:37327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:37327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:31.940043 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:31.941485 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:31.955137 19905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:31.955003 19906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:31.954958 19908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:31.956355 19899 server_base.cc:1034] running on GCE node
I20260811 03:10:31.957340 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:31.959021 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:31.960256 19899 hybrid_clock.cc:648] HybridClock initialized: now 1786417831960210 us; error 85 us; skew 500 ppm
I20260811 03:10:31.960637 19899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:31 dist-test-slave-8tdl krb5kdc[19886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:31.964296 19899 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:10:31.966150 19899 webserver.cc:460] Webserver started at http://127.27.109.62:39107/ using document root <none> and password file <none>
I20260811 03:10:31.967044 19899 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:31.967204 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:31.967574 19899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:31.971084 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "7c221ee951794ade863a7d0907fd0914"
format_stamp: "Formatted at 2026-08-11 03:10:31 on dist-test-slave-8tdl"
server_key: "58a289ab824bed34fe8c1797c36549a2"
server_key_iv: "4efd509ac1e3dcb3435a45e0cc4b0d50"
server_key_version: "encryptionkey@0"
I20260811 03:10:31.971952 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "7c221ee951794ade863a7d0907fd0914"
format_stamp: "Formatted at 2026-08-11 03:10:31 on dist-test-slave-8tdl"
server_key: "58a289ab824bed34fe8c1797c36549a2"
server_key_iv: "4efd509ac1e3dcb3435a45e0cc4b0d50"
server_key_version: "encryptionkey@0"
I20260811 03:10:31.977886 19899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260811 03:10:31.982299 19916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:31.984210 19899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260811 03:10:31.984562 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "7c221ee951794ade863a7d0907fd0914"
format_stamp: "Formatted at 2026-08-11 03:10:31 on dist-test-slave-8tdl"
server_key: "58a289ab824bed34fe8c1797c36549a2"
server_key_iv: "4efd509ac1e3dcb3435a45e0cc4b0d50"
server_key_version: "encryptionkey@0"
I20260811 03:10:31.984781 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:32.002286 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:32.007467 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:32.007818 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:32.029873 19899 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:37327
I20260811 03:10:32.029906 19975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:37327 every 8 connection(s)
I20260811 03:10:32.031560 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:32.036167 19976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:32.041388 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 19899
I20260811 03:10:32.041601 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:32.042181 28084 external_mini_cluster.cc:1428] Setting key 7288a381a861c71ed4a63dbde94f6388
I20260811 03:10:32.049873 19976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914: Bootstrap starting.
I20260811 03:10:32.054250 19976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:32.055681 19976 log.cc:824] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:32.060051 19976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914: No bootstrap required, opened a new log
Aug 11 03:10:32 dist-test-slave-8tdl krb5kdc[19886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:10:32.072237 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:32.044428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58759 (local address 127.27.109.62:37327)
0811 03:10:32.045377 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:32.045391 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:32.045441 (+ 50us) server_negotiation.cc:407] Connection header received
0811 03:10:32.048128 (+ 2687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:32.048160 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:32.048650 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:32.049066 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:32.051285 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:32.052791 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:32.054809 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:32.055642 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:32.055812 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:32.061629 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:32.061642 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:32.061665 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:32.061735 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:32.066472 (+ 4737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:32.067630 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:32.067639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:32.067649 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:32.067905 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:32.068436 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:32.068441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:32.068445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:32.069309 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:32.069701 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:32.070150 (+ 449us) server_negotiation.cc:299] Negotiation successful
0811 03:10:32.070595 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":466,"threads_started":1}
I20260811 03:10:32.072237 19976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c221ee951794ade863a7d0907fd0914" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37327 } }
I20260811 03:10:32.072829 19976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:32.072993 19976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c221ee951794ade863a7d0907fd0914, State: Initialized, Role: FOLLOWER
I20260811 03:10:32.073638 19976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c221ee951794ade863a7d0907fd0914" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37327 } }
I20260811 03:10:32.073931 19976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:32.074085 19976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:32.074255 19976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:32.077085 19976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c221ee951794ade863a7d0907fd0914" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37327 } }
I20260811 03:10:32.077692 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:32.077787 19976 leader_election.cc:302] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c221ee951794ade863a7d0907fd0914; no voters:
I20260811 03:10:32.078514 19976 leader_election.cc:288] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:32.078665 19981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:32.080224 19981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [term 1 LEADER]: Becoming Leader. State: Replica: 7c221ee951794ade863a7d0907fd0914, State: Running, Role: LEADER
I20260811 03:10:32.080899 19981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c221ee951794ade863a7d0907fd0914" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37327 } }
I20260811 03:10:32.081149 19976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:32.085191 19982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c221ee951794ade863a7d0907fd0914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c221ee951794ade863a7d0907fd0914" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37327 } } }
I20260811 03:10:32.085659 19982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:32.086325 19983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c221ee951794ade863a7d0907fd0914. Latest consensus state: current_term: 1 leader_uuid: "7c221ee951794ade863a7d0907fd0914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c221ee951794ade863a7d0907fd0914" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 37327 } } }
I20260811 03:10:32.086802 19983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:32.097165 19996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:32.105515 19996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:32.120352 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:32.085372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33033 (local address 127.27.109.62:37327)
0811 03:10:32.090646 (+ 5274us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:32.090655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:32.095538 (+ 4883us) server_negotiation.cc:407] Connection header received
0811 03:10:32.095944 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:32.095956 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:32.096081 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:32.096180 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:32.099462 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:32.101205 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:32.104378 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:32.105451 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:32.105611 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:32.110464 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:32.110476 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:32.110484 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:32.110557 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:32.114880 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:32.116454 (+ 1574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:32.116464 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:32.116470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:32.116680 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:32.117506 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:32.117514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:32.117520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:32.118101 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:32.118461 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:32.119467 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0811 03:10:32.119781 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4917}
I20260811 03:10:32.128064 19996 catalog_manager.cc:1347] Generated new cluster ID: 6bad721c4be04c3780dbd5234cb8ef6d
I20260811 03:10:32.128350 19996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:32.142820 20008 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 03:10:32.148326 19996 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:32.150276 19996 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:32.160456 19996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c221ee951794ade863a7d0907fd0914: Generated new TSK 0
I20260811 03:10:32.161441 19996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:32.169919 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 19899
I20260811 03:10:32.171658 19970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:32.268436 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 19899
W20260811 03:10:32.291956 19884 connection.cc:535] client connection to 127.27.109.62:37327 recv error: Network error: failed to read from TLS socket (remote: 127.27.109.62:37327): Connection reset by peer (error 104)
2026-08-11T03:10:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:32 dist-test-slave-8tdl krb5kdc[20012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:32 dist-test-slave-8tdl krb5kdc[20012](info): set up 2 sockets
Aug 11 03:10:32 dist-test-slave-8tdl krb5kdc[20012](info): commencing operation
krb5kdc: starting...
W20260811 03:10:33.526718 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:33 dist-test-slave-8tdl krb5kdc[20012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:10:33.910634 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:44639
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:44639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:34.311270 20026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:34.311671 20026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:34.311839 20026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:34.319991 20026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:34.320251 20026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:34.320377 20026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:34.320505 20026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:34.320600 20026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:34.332286 20026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:44639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:44639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:34.333874 20026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:34.336112 20026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:34.350069 20031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:34.350112 20034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:34.350116 20032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:34.351318 20026 server_base.cc:1034] running on GCE node
I20260811 03:10:34.352406 20026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:34.354110 20026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:34.355391 20026 hybrid_clock.cc:648] HybridClock initialized: now 1786417834355343 us; error 49 us; skew 500 ppm
I20260811 03:10:34.355855 20026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:34 dist-test-slave-8tdl krb5kdc[20012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417834, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:34.359831 20026 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:10:34.361882 20026 webserver.cc:460] Webserver started at http://127.27.109.62:38057/ using document root <none> and password file <none>
I20260811 03:10:34.362942 20026 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:34.363133 20026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:34.363534 20026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:34.367439 20026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "aaa28468cabc4884a9e46bd5eae8636c"
format_stamp: "Formatted at 2026-08-11 03:10:34 on dist-test-slave-8tdl"
server_key: "fb5237b2bcf6d73447f43047f428c951"
server_key_iv: "55af16931adaf21b07e693dcf43585e8"
server_key_version: "encryptionkey@0"
I20260811 03:10:34.368496 20026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "aaa28468cabc4884a9e46bd5eae8636c"
format_stamp: "Formatted at 2026-08-11 03:10:34 on dist-test-slave-8tdl"
server_key: "fb5237b2bcf6d73447f43047f428c951"
server_key_iv: "55af16931adaf21b07e693dcf43585e8"
server_key_version: "encryptionkey@0"
I20260811 03:10:34.374969 20026 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260811 03:10:34.379035 20041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:34.380525 20026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:10:34.380716 20026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "aaa28468cabc4884a9e46bd5eae8636c"
format_stamp: "Formatted at 2026-08-11 03:10:34 on dist-test-slave-8tdl"
server_key: "fb5237b2bcf6d73447f43047f428c951"
server_key_iv: "55af16931adaf21b07e693dcf43585e8"
server_key_version: "encryptionkey@0"
I20260811 03:10:34.380937 20026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:34.404990 20026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:34.410790 20026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:34.411173 20026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:34.437057 20026 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:44639
I20260811 03:10:34.437054 20100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:44639 every 8 connection(s)
I20260811 03:10:34.438828 20026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:34.441910 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 20026
I20260811 03:10:34.442104 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:34.442588 28084 external_mini_cluster.cc:1428] Setting key d1781d9896dcfd1e6dde1a6dde02e37b
I20260811 03:10:34.443145 20101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:34.457782 20101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c: Bootstrap starting.
I20260811 03:10:34.460592 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:34.444656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37219 (local address 127.27.109.62:44639)
0811 03:10:34.447941 (+ 3285us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:34.447956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:34.448001 (+ 45us) server_negotiation.cc:407] Connection header received
0811 03:10:34.450458 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:34.450490 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:34.451112 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:34.451587 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:34.452220 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:34.453805 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:34.455419 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:34.456207 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:34.456446 (+ 239us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:34.457890 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:34.457899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:34.457916 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:34.458554 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:34.458652 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:34.458734 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:34.458914 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:34.459230 (+ 316us) server_negotiation.cc:299] Negotiation successful
0811 03:10:34.459420 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3043,"thread_start_us":2825,"threads_started":1}
I20260811 03:10:34.461956 20101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:34.463586 20101 log.cc:824] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:34.465186 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:34.467113 20101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c: No bootstrap required, opened a new log
I20260811 03:10:34.477191 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:34.470261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36847 (local address 127.27.109.62:44639)
0811 03:10:34.470440 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:34.470445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:34.470988 (+ 543us) server_negotiation.cc:407] Connection header received
0811 03:10:34.471253 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:34.471257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:34.471333 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:34.471443 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:34.472052 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:34.472881 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:34.474315 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:34.474914 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:34.475073 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:34.476353 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:34.476358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:34.476362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:34.476382 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:34.476440 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:34.476457 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:34.476605 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:34.476807 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 03:10:34.476876 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 03:10:34.476831 20101 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa28468cabc4884a9e46bd5eae8636c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44639 } }
I20260811 03:10:34.477530 20101 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:34.477684 20101 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa28468cabc4884a9e46bd5eae8636c, State: Initialized, Role: FOLLOWER
I20260811 03:10:34.478528 20101 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa28468cabc4884a9e46bd5eae8636c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44639 } }
I20260811 03:10:34.478912 20101 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:34.479113 20101 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:34.479312 20101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:34.481945 20101 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa28468cabc4884a9e46bd5eae8636c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44639 } }
I20260811 03:10:34.482791 20101 leader_election.cc:302] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaa28468cabc4884a9e46bd5eae8636c; no voters:
I20260811 03:10:34.483523 20101 leader_election.cc:288] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:34.483682 20111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:34.484737 20111 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [term 1 LEADER]: Becoming Leader. State: Replica: aaa28468cabc4884a9e46bd5eae8636c, State: Running, Role: LEADER
I20260811 03:10:34.485514 20111 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa28468cabc4884a9e46bd5eae8636c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44639 } }
I20260811 03:10:34.486361 20101 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:34.488692 20112 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaa28468cabc4884a9e46bd5eae8636c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa28468cabc4884a9e46bd5eae8636c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44639 } } }
I20260811 03:10:34.489200 20112 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:34.490046 20113 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaa28468cabc4884a9e46bd5eae8636c. Latest consensus state: current_term: 1 leader_uuid: "aaa28468cabc4884a9e46bd5eae8636c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa28468cabc4884a9e46bd5eae8636c" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 44639 } } }
I20260811 03:10:34.490311 20118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:34.490497 20113 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:34.496801 20118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:34.507043 20118 catalog_manager.cc:1347] Generated new cluster ID: 11b88e9d26194f0fa956ef1866bd9ab6
I20260811 03:10:34.507200 20118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:34.542032 20118 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:34.544207 20118 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:34.554227 20118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaa28468cabc4884a9e46bd5eae8636c: Generated new TSK 0
I20260811 03:10:34.555536 20118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:34.572359 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20026
W20260811 03:10:34.573828 20096 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37219
W20260811 03:10:34.574769 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20026: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:34.574951 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 20026
W20260811 03:10:34.598744 20010 connection.cc:535] client connection to 127.27.109.62:44639 recv error: Network error: failed to read from TLS socket (remote: 127.27.109.62:44639): Connection reset by peer (error 104)
2026-08-11T03:10:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:34 dist-test-slave-8tdl krb5kdc[20132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:34 dist-test-slave-8tdl krb5kdc[20132](info): set up 2 sockets
Aug 11 03:10:34 dist-test-slave-8tdl krb5kdc[20132](info): commencing operation
krb5kdc: starting...
W20260811 03:10:35.826455 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:36 dist-test-slave-8tdl krb5kdc[20132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:10:36.206053 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38733
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:36.574810 20145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:36.575239 20145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:36.575413 20145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:36.583415 20145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:36.583612 20145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:36.583722 20145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:36.583833 20145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:36.583920 20145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:36.594388 20145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:36.595901 20145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:36.597286 20145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:36.610455 20150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:36.610435 20151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:36.610984 20153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:36.611150 20145 server_base.cc:1034] running on GCE node
I20260811 03:10:36.612187 20145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:36.613783 20145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:36.615068 20145 hybrid_clock.cc:648] HybridClock initialized: now 1786417836614999 us; error 55 us; skew 500 ppm
I20260811 03:10:36.615496 20145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:36 dist-test-slave-8tdl krb5kdc[20132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417836, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:36.619138 20145 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:10:36.620867 20145 webserver.cc:460] Webserver started at http://127.27.109.62:43967/ using document root <none> and password file <none>
I20260811 03:10:36.621627 20145 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:36.621781 20145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:36.622116 20145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:36.625451 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "172fde37a35942cfa85c1f54c6d94ce9"
format_stamp: "Formatted at 2026-08-11 03:10:36 on dist-test-slave-8tdl"
server_key: "c8063e9e7e5917dce66e8ccac9cfbdd1"
server_key_iv: "ceb2063cdc90d66449c8fbb997c7f324"
server_key_version: "encryptionkey@0"
I20260811 03:10:36.626305 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "172fde37a35942cfa85c1f54c6d94ce9"
format_stamp: "Formatted at 2026-08-11 03:10:36 on dist-test-slave-8tdl"
server_key: "c8063e9e7e5917dce66e8ccac9cfbdd1"
server_key_iv: "ceb2063cdc90d66449c8fbb997c7f324"
server_key_version: "encryptionkey@0"
I20260811 03:10:36.632061 20145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 03:10:36.635667 20160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:36.637249 20145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 03:10:36.637470 20145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "172fde37a35942cfa85c1f54c6d94ce9"
format_stamp: "Formatted at 2026-08-11 03:10:36 on dist-test-slave-8tdl"
server_key: "c8063e9e7e5917dce66e8ccac9cfbdd1"
server_key_iv: "ceb2063cdc90d66449c8fbb997c7f324"
server_key_version: "encryptionkey@0"
I20260811 03:10:36.637703 20145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:36.643792 20145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:36.648761 20145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:36.649116 20145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:36.670388 20145 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:38733
I20260811 03:10:36.670449 20217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:38733 every 8 connection(s)
I20260811 03:10:36.672014 20145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:36.675980 20218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:36.682337 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 20145
I20260811 03:10:36.682566 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:36.683162 28084 external_mini_cluster.cc:1428] Setting key e22c14b454733df6cc44a6e0e3e597fb
I20260811 03:10:36.688252 20218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9: Bootstrap starting.
I20260811 03:10:36.692514 20218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:36.692966 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:36.685347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42323 (local address 127.27.109.62:38733)
0811 03:10:36.686291 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:36.686306 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:36.686358 (+ 52us) server_negotiation.cc:407] Connection header received
0811 03:10:36.688847 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:36.688880 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:36.689346 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:36.689807 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:36.690967 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:36.690978 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:36.690995 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:36.691013 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:36.691082 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:36.691112 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:36.691337 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:36.691459 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 03:10:36.691668 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":416,"threads_started":1}
I20260811 03:10:36.693835 20218 log.cc:824] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:36.697453 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:36.697674 20218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9: No bootstrap required, opened a new log
I20260811 03:10:36.705125 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:36.702246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41923 (local address 127.27.109.62:38733)
0811 03:10:36.702494 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:36.702503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:36.703066 (+ 563us) server_negotiation.cc:407] Connection header received
0811 03:10:36.703171 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:36.703175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:36.703229 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:36.703323 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:36.704350 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:36.704354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:36.704358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:36.704372 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:36.704410 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:36.704424 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:36.704524 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:36.704659 (+ 135us) server_negotiation.cc:299] Negotiation successful
0811 03:10:36.704713 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 03:10:36.707314 20218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fde37a35942cfa85c1f54c6d94ce9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38733 } }
I20260811 03:10:36.707746 20218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:36.707887 20218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172fde37a35942cfa85c1f54c6d94ce9, State: Initialized, Role: FOLLOWER
I20260811 03:10:36.708489 20218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fde37a35942cfa85c1f54c6d94ce9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38733 } }
I20260811 03:10:36.708763 20218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:36.708930 20218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:36.709142 20218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:36.711889 20218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fde37a35942cfa85c1f54c6d94ce9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38733 } }
I20260811 03:10:36.712384 20218 leader_election.cc:302] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 172fde37a35942cfa85c1f54c6d94ce9; no voters:
I20260811 03:10:36.712980 20218 leader_election.cc:288] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:36.713161 20228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:36.713891 20228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [term 1 LEADER]: Becoming Leader. State: Replica: 172fde37a35942cfa85c1f54c6d94ce9, State: Running, Role: LEADER
I20260811 03:10:36.714459 20228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fde37a35942cfa85c1f54c6d94ce9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38733 } }
I20260811 03:10:36.715284 20218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:36.717707 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 172fde37a35942cfa85c1f54c6d94ce9. Latest consensus state: current_term: 1 leader_uuid: "172fde37a35942cfa85c1f54c6d94ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fde37a35942cfa85c1f54c6d94ce9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38733 } } }
I20260811 03:10:36.718284 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:36.718146 20229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "172fde37a35942cfa85c1f54c6d94ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172fde37a35942cfa85c1f54c6d94ce9" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38733 } } }
I20260811 03:10:36.718485 20229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:36.720270 20235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:36.726387 20235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:36.736164 20235 catalog_manager.cc:1347] Generated new cluster ID: 06772a82d7da46f895c042e72b2d632f
I20260811 03:10:36.736362 20235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:36.752993 20235 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:36.753563 20235 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:36.774783 20235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 172fde37a35942cfa85c1f54c6d94ce9: Generated new TSK 0
I20260811 03:10:36.776075 20235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:36.787278 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20145
W20260811 03:10:36.789031 20212 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42323
W20260811 03:10:36.789878 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20145: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:36.790110 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 20145
2026-08-11T03:10:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:36 dist-test-slave-8tdl krb5kdc[20249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:36 dist-test-slave-8tdl krb5kdc[20249](info): set up 2 sockets
Aug 11 03:10:36 dist-test-slave-8tdl krb5kdc[20249](info): commencing operation
krb5kdc: starting...
W20260811 03:10:38.032253 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.002s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:38 dist-test-slave-8tdl krb5kdc[20249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:10:38.398224 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34183
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:34183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:38.767563 20262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:38.767895 20262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:38.768090 20262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:38.775915 20262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:38.776139 20262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:38.776239 20262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:38.776338 20262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:38.776408 20262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:38.788125 20262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:34183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:34183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:38.789657 20262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:38.791424 20262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:38.804482 20268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:38.804602 20270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:38.804469 20267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:38.805583 20262 server_base.cc:1034] running on GCE node
I20260811 03:10:38.806461 20262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:38.808179 20262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:38.809451 20262 hybrid_clock.cc:648] HybridClock initialized: now 1786417838809409 us; error 54 us; skew 500 ppm
I20260811 03:10:38.809886 20262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:38 dist-test-slave-8tdl krb5kdc[20249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:38.813683 20262 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:10:38.815665 20262 webserver.cc:460] Webserver started at http://127.27.109.62:34657/ using document root <none> and password file <none>
I20260811 03:10:38.816510 20262 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:38.816686 20262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:38.817075 20262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:38.821566 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "61f60ef2dd8f4844a0b79008a0170812"
format_stamp: "Formatted at 2026-08-11 03:10:38 on dist-test-slave-8tdl"
server_key: "23a12d8fd0c38aafbe388a4608c6737d"
server_key_iv: "c004e539dbfa3c088e84f4b38cfa9c07"
server_key_version: "encryptionkey@0"
I20260811 03:10:38.822602 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "61f60ef2dd8f4844a0b79008a0170812"
format_stamp: "Formatted at 2026-08-11 03:10:38 on dist-test-slave-8tdl"
server_key: "23a12d8fd0c38aafbe388a4608c6737d"
server_key_iv: "c004e539dbfa3c088e84f4b38cfa9c07"
server_key_version: "encryptionkey@0"
I20260811 03:10:38.828953 20262 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:10:38.833469 20278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:38.835233 20262 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 03:10:38.835480 20262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "61f60ef2dd8f4844a0b79008a0170812"
format_stamp: "Formatted at 2026-08-11 03:10:38 on dist-test-slave-8tdl"
server_key: "23a12d8fd0c38aafbe388a4608c6737d"
server_key_iv: "c004e539dbfa3c088e84f4b38cfa9c07"
server_key_version: "encryptionkey@0"
I20260811 03:10:38.835737 20262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:38.855269 20262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:38.860846 20262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:38.861205 20262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:38.882208 20262 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:34183
I20260811 03:10:38.882269 20334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:34183 every 8 connection(s)
I20260811 03:10:38.883822 20262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:38.886783 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 20262
I20260811 03:10:38.887004 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:38.887603 28084 external_mini_cluster.cc:1428] Setting key 098b07a5fae9a0859412a06c22ec5957
I20260811 03:10:38.889279 20335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 03:10:38 dist-test-slave-8tdl krb5kdc[20249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:10:38.906838 20335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812: Bootstrap starting.
I20260811 03:10:38.910849 20335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:38.912360 20335 log.cc:824] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:38.915805 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:38.889912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60859 (local address 127.27.109.62:34183)
0811 03:10:38.890929 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:38.890943 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:38.890999 (+ 56us) server_negotiation.cc:407] Connection header received
0811 03:10:38.892855 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:38.892884 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:38.893400 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:38.893814 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:38.896218 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:38.897914 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:38.900029 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:38.900784 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:38.900946 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:38.906561 (+ 5615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:38.906571 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:38.906590 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:38.906640 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:38.910474 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:38.911654 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:38.911663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:38.911669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:38.911877 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:38.912383 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:38.912387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:38.912389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:38.913224 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:38.913599 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:38.914162 (+ 563us) server_negotiation.cc:299] Negotiation successful
0811 03:10:38.914654 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":475,"threads_started":1}
I20260811 03:10:38.916178 20335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812: No bootstrap required, opened a new log
I20260811 03:10:38.920099 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:38.926571 20335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f60ef2dd8f4844a0b79008a0170812" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34183 } }
I20260811 03:10:38.927237 20335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:38.927384 20335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f60ef2dd8f4844a0b79008a0170812, State: Initialized, Role: FOLLOWER
I20260811 03:10:38.928121 20335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f60ef2dd8f4844a0b79008a0170812" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34183 } }
I20260811 03:10:38.928409 20335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:38.928617 20335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:38.928822 20335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:38.931962 20335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f60ef2dd8f4844a0b79008a0170812" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34183 } }
I20260811 03:10:38.932664 20335 leader_election.cc:302] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f60ef2dd8f4844a0b79008a0170812; no voters:
I20260811 03:10:38.933411 20335 leader_election.cc:288] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:38.933583 20345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:38.935750 20335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:38.937036 20345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [term 1 LEADER]: Becoming Leader. State: Replica: 61f60ef2dd8f4844a0b79008a0170812, State: Running, Role: LEADER
I20260811 03:10:38.937662 20345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f60ef2dd8f4844a0b79008a0170812" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34183 } }
I20260811 03:10:38.940703 20347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f60ef2dd8f4844a0b79008a0170812. Latest consensus state: current_term: 1 leader_uuid: "61f60ef2dd8f4844a0b79008a0170812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f60ef2dd8f4844a0b79008a0170812" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34183 } } }
I20260811 03:10:38.941191 20347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:38.941792 20346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61f60ef2dd8f4844a0b79008a0170812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f60ef2dd8f4844a0b79008a0170812" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 34183 } } }
I20260811 03:10:38.942121 20346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:38.948568 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:38.926023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39307 (local address 127.27.109.62:34183)
0811 03:10:38.926246 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:38.926253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:38.926814 (+ 561us) server_negotiation.cc:407] Connection header received
0811 03:10:38.926983 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:38.926988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:38.927187 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:38.927325 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:38.929189 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:38.930445 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:38.931985 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:38.932648 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:38.932798 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:38.936768 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:38.936777 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:38.936783 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:38.936836 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:38.942651 (+ 5815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:38.944105 (+ 1454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:38.944114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:38.944121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:38.944396 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:38.945066 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:38.945074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:38.945080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:38.945613 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:38.947419 (+ 1806us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:38.947627 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 03:10:38.947982 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:10:38.956305 20350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:38.961593 20350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:38.970793 20350 catalog_manager.cc:1347] Generated new cluster ID: c8108d4c0d9b4efe88129fa276b9beeb
I20260811 03:10:38.970940 20350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:38.987494 20350 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:38.989040 20350 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:39.003007 20350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61f60ef2dd8f4844a0b79008a0170812: Generated new TSK 0
I20260811 03:10:39.003983 20350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:39.015745 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20262
I20260811 03:10:39.017382 20330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:39.113511 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 20262
2026-08-11T03:10:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:39 dist-test-slave-8tdl krb5kdc[20370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:39 dist-test-slave-8tdl krb5kdc[20370](info): set up 2 sockets
Aug 11 03:10:39 dist-test-slave-8tdl krb5kdc[20370](info): commencing operation
krb5kdc: starting...
W20260811 03:10:40.368080 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.002s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:40 dist-test-slave-8tdl krb5kdc[20370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:10:40.736156 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:45977
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:41.112030 20383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:41.112370 20383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:41.112532 20383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:41.120157 20383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:41.120291 20383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:41.120373 20383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:41.120484 20383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:41.120599 20383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:41.131359 20383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:41.132814 20383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:41.134115 20383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:41.146991 20389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:41.147100 20390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:41.147100 20392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:41.147975 20383 server_base.cc:1034] running on GCE node
I20260811 03:10:41.148911 20383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:41.150446 20383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:41.151700 20383 hybrid_clock.cc:648] HybridClock initialized: now 1786417841151648 us; error 50 us; skew 500 ppm
I20260811 03:10:41.152144 20383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:41 dist-test-slave-8tdl krb5kdc[20370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417841, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:41.155566 20383 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:10:41.157272 20383 webserver.cc:460] Webserver started at http://127.27.109.62:33843/ using document root <none> and password file <none>
I20260811 03:10:41.158377 20383 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:41.158598 20383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:41.158990 20383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:41.162261 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "3b9226a99c6742d4a29b75ab96cf1517"
format_stamp: "Formatted at 2026-08-11 03:10:41 on dist-test-slave-8tdl"
server_key: "cb4a9df49a88adaca476550fb906cce7"
server_key_iv: "aae34f038e3224e20b94b9d6f5ed5fb5"
server_key_version: "encryptionkey@0"
I20260811 03:10:41.163133 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "3b9226a99c6742d4a29b75ab96cf1517"
format_stamp: "Formatted at 2026-08-11 03:10:41 on dist-test-slave-8tdl"
server_key: "cb4a9df49a88adaca476550fb906cce7"
server_key_iv: "aae34f038e3224e20b94b9d6f5ed5fb5"
server_key_version: "encryptionkey@0"
I20260811 03:10:41.168841 20383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:10:41.172746 20401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:41.174140 20383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 03:10:41.174346 20383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "3b9226a99c6742d4a29b75ab96cf1517"
format_stamp: "Formatted at 2026-08-11 03:10:41 on dist-test-slave-8tdl"
server_key: "cb4a9df49a88adaca476550fb906cce7"
server_key_iv: "aae34f038e3224e20b94b9d6f5ed5fb5"
server_key_version: "encryptionkey@0"
I20260811 03:10:41.174583 20383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:41.187628 20383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:41.192009 20383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:41.192344 20383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:41.211591 20383 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:45977
I20260811 03:10:41.211632 20455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:45977 every 8 connection(s)
I20260811 03:10:41.213194 20383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:41.213584 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 20383
I20260811 03:10:41.213788 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:41.214469 28084 external_mini_cluster.cc:1428] Setting key e160b7deb0a287868e5c7f25932ce6cd
I20260811 03:10:41.218055 20456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:41.228147 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:41.216479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51317 (local address 127.27.109.62:45977)
0811 03:10:41.217270 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:41.217282 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:41.217324 (+ 42us) server_negotiation.cc:407] Connection header received
0811 03:10:41.219181 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:41.219204 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:41.219596 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:41.219986 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:41.220530 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:41.221887 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:41.223300 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:41.223978 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:41.224180 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:41.225502 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:41.225510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:41.225535 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:41.226121 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:41.226203 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:41.226229 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:41.226429 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:41.226671 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 03:10:41.226964 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":353,"threads_started":1}
I20260811 03:10:41.229966 20456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517: Bootstrap starting.
I20260811 03:10:41.232333 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:41.233659 20456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:41.234758 20456 log.cc:824] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:41.237879 20456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517: No bootstrap required, opened a new log
I20260811 03:10:41.243956 20459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:41.237005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35033 (local address 127.27.109.62:45977)
0811 03:10:41.237221 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:41.237227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:41.237752 (+ 525us) server_negotiation.cc:407] Connection header received
0811 03:10:41.237900 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:41.237909 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:41.237992 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:41.238116 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:41.238588 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:41.239694 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:41.241023 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:41.241798 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:41.241930 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:41.243130 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:41.243138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:41.243145 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:41.243172 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:41.243226 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:41.243248 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:41.243348 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:41.243471 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 03:10:41.243574 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 03:10:41.247978 20456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9226a99c6742d4a29b75ab96cf1517" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45977 } }
I20260811 03:10:41.248438 20456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:41.248570 20456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b9226a99c6742d4a29b75ab96cf1517, State: Initialized, Role: FOLLOWER
I20260811 03:10:41.249184 20456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9226a99c6742d4a29b75ab96cf1517" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45977 } }
I20260811 03:10:41.249439 20456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:41.249610 20456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:41.249810 20456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:41.252760 20456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9226a99c6742d4a29b75ab96cf1517" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45977 } }
I20260811 03:10:41.253314 20456 leader_election.cc:302] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b9226a99c6742d4a29b75ab96cf1517; no voters:
I20260811 03:10:41.253968 20456 leader_election.cc:288] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:41.254165 20466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:41.255158 20466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [term 1 LEADER]: Becoming Leader. State: Replica: 3b9226a99c6742d4a29b75ab96cf1517, State: Running, Role: LEADER
I20260811 03:10:41.256022 20466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9226a99c6742d4a29b75ab96cf1517" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45977 } }
I20260811 03:10:41.256395 20456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:41.259765 20467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b9226a99c6742d4a29b75ab96cf1517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9226a99c6742d4a29b75ab96cf1517" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45977 } } }
I20260811 03:10:41.259729 20468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b9226a99c6742d4a29b75ab96cf1517. Latest consensus state: current_term: 1 leader_uuid: "3b9226a99c6742d4a29b75ab96cf1517" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b9226a99c6742d4a29b75ab96cf1517" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 45977 } } }
I20260811 03:10:41.260262 20467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:41.260262 20468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:41.261950 20475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:41.269575 20475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:41.279552 20475 catalog_manager.cc:1347] Generated new cluster ID: e9cee72b143f4059946b1d238ca01eed
I20260811 03:10:41.279690 20475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:41.304772 20475 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:41.306483 20475 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:41.324227 20475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b9226a99c6742d4a29b75ab96cf1517: Generated new TSK 0
I20260811 03:10:41.325239 20475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:41.339401 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20383
W20260811 03:10:41.340988 20450 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51317
W20260811 03:10:41.341787 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20383: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:41.341974 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 20383
2026-08-11T03:10:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:41 dist-test-slave-8tdl krb5kdc[20489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:41 dist-test-slave-8tdl krb5kdc[20489](info): set up 2 sockets
Aug 11 03:10:41 dist-test-slave-8tdl krb5kdc[20489](info): commencing operation
krb5kdc: starting...
W20260811 03:10:42.603196 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.006s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:42 dist-test-slave-8tdl krb5kdc[20489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:10:42.982393 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46233
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:46233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:43.351181 20504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:43.351516 20504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:43.351693 20504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:43.359601 20504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:43.359735 20504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:43.359839 20504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:43.359951 20504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:43.360034 20504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:43.371804 20504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:46233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:43.373634 20504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:43.375342 20504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:43.389693 20510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:43.389742 20509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:43.389787 20512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:43.390882 20504 server_base.cc:1034] running on GCE node
I20260811 03:10:43.391873 20504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:43.393653 20504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:43.394933 20504 hybrid_clock.cc:648] HybridClock initialized: now 1786417843394884 us; error 48 us; skew 500 ppm
I20260811 03:10:43.395430 20504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:43 dist-test-slave-8tdl krb5kdc[20489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417843, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:43.399241 20504 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:10:43.401294 20504 webserver.cc:460] Webserver started at http://127.27.109.62:44073/ using document root <none> and password file <none>
I20260811 03:10:43.402101 20504 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:43.402264 20504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:43.402611 20504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:43.405928 20504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "a3623f0d94f94a678e9225e79a34f729"
format_stamp: "Formatted at 2026-08-11 03:10:43 on dist-test-slave-8tdl"
server_key: "9ea339a83d35c11cc5de162060b62720"
server_key_iv: "7a0f0b2137622bfc453c0995accbcc2c"
server_key_version: "encryptionkey@0"
I20260811 03:10:43.406893 20504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "a3623f0d94f94a678e9225e79a34f729"
format_stamp: "Formatted at 2026-08-11 03:10:43 on dist-test-slave-8tdl"
server_key: "9ea339a83d35c11cc5de162060b62720"
server_key_iv: "7a0f0b2137622bfc453c0995accbcc2c"
server_key_version: "encryptionkey@0"
I20260811 03:10:43.412683 20504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 03:10:43.416705 20519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:43.418342 20504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 03:10:43.418555 20504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "a3623f0d94f94a678e9225e79a34f729"
format_stamp: "Formatted at 2026-08-11 03:10:43 on dist-test-slave-8tdl"
server_key: "9ea339a83d35c11cc5de162060b62720"
server_key_iv: "7a0f0b2137622bfc453c0995accbcc2c"
server_key_version: "encryptionkey@0"
I20260811 03:10:43.418859 20504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:43.424999 20504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:43.429765 20504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:43.430117 20504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:43.450392 20504 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:46233
I20260811 03:10:43.450431 20576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:46233 every 8 connection(s)
I20260811 03:10:43.451972 20504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:43.455943 20577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:43.460392 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 20504
I20260811 03:10:43.460598 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:43.461195 28084 external_mini_cluster.cc:1428] Setting key b4891382171feb36eff43c0a4a9c0d0a
I20260811 03:10:43.468411 20577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729: Bootstrap starting.
I20260811 03:10:43.470654 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:43.463240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51551 (local address 127.27.109.62:46233)
0811 03:10:43.464103 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:43.464115 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:43.464158 (+ 43us) server_negotiation.cc:407] Connection header received
0811 03:10:43.466578 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:43.466604 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:43.467065 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:43.467396 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:43.468559 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:43.468570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:43.468590 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:43.468615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:43.468697 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:43.468729 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:43.469300 (+ 571us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:43.469429 (+ 129us) server_negotiation.cc:299] Negotiation successful
0811 03:10:43.469663 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":369,"threads_started":1}
I20260811 03:10:43.472546 20577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:43.473958 20577 log.cc:824] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:43.475095 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:43.477090 20577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729: No bootstrap required, opened a new log
I20260811 03:10:43.482627 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:43.479840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33715 (local address 127.27.109.62:46233)
0811 03:10:43.480052 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:43.480065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:43.480467 (+ 402us) server_negotiation.cc:407] Connection header received
0811 03:10:43.480626 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:43.480632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:43.480705 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:43.480826 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:43.481779 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:43.481785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:43.481791 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 03:10:43.481816 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:43.481873 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 03:10:43.481899 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:43.482026 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:43.482217 (+ 191us) server_negotiation.cc:299] Negotiation successful
0811 03:10:43.482289 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:10:43.486522 20577 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3623f0d94f94a678e9225e79a34f729" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46233 } }
I20260811 03:10:43.487051 20577 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 03:10:43.487195 20577 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3623f0d94f94a678e9225e79a34f729, State: Initialized, Role: FOLLOWER
I20260811 03:10:43.487902 20577 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3623f0d94f94a678e9225e79a34f729" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46233 } }
I20260811 03:10:43.488193 20577 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 03:10:43.488353 20577 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 03:10:43.488548 20577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:43.490968 20577 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3623f0d94f94a678e9225e79a34f729" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46233 } }
I20260811 03:10:43.491532 20577 leader_election.cc:302] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3623f0d94f94a678e9225e79a34f729; no voters:
I20260811 03:10:43.492506 20577 leader_election.cc:288] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 03:10:43.492640 20587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:43.495034 20577 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:43.495446 20587 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [term 1 LEADER]: Becoming Leader. State: Replica: a3623f0d94f94a678e9225e79a34f729, State: Running, Role: LEADER
I20260811 03:10:43.496012 20587 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3623f0d94f94a678e9225e79a34f729" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46233 } }
I20260811 03:10:43.499408 20588 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3623f0d94f94a678e9225e79a34f729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3623f0d94f94a678e9225e79a34f729" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46233 } } }
I20260811 03:10:43.500428 20588 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:43.499867 20589 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3623f0d94f94a678e9225e79a34f729. Latest consensus state: current_term: 1 leader_uuid: "a3623f0d94f94a678e9225e79a34f729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3623f0d94f94a678e9225e79a34f729" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 46233 } } }
I20260811 03:10:43.500926 20589 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:43.502177 20599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:43.507194 20599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:43.516381 20599 catalog_manager.cc:1347] Generated new cluster ID: d9c5c1c4889044218ae252f3c4d37785
I20260811 03:10:43.516526 20599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:43.544183 20599 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:43.544781 20599 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:43.553263 20599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3623f0d94f94a678e9225e79a34f729: Generated new TSK 0
I20260811 03:10:43.554473 20599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 03:10:43.564620 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20504
W20260811 03:10:43.565904 20572 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51551
W20260811 03:10:43.566761 28084 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20504: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 03:10:43.566915 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 20504
2026-08-11T03:10:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2225 ms)
[----------] 18 tests from AuthTokenIssuingTest (40804 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:43 dist-test-slave-8tdl krb5kdc[20611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:43 dist-test-slave-8tdl krb5kdc[20611](info): set up 2 sockets
Aug 11 03:10:43 dist-test-slave-8tdl krb5kdc[20611](info): commencing operation
krb5kdc: starting...
W20260811 03:10:44.803541 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.009s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:45 dist-test-slave-8tdl krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:10:45.191798 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36491
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36491,127.27.109.61:38547,127.27.109.60:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:45.567981 20625 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 03:10:45.568435 20625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:45.568568 20625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:45.568718 20625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:45.576498 20625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:45.576617 20625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:45.576725 20625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:45.576833 20625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:45.576930 20625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:45.587479 20625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36491,127.27.109.61:38547,127.27.109.60:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:36491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:45.589067 20625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:45.590509 20625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:45.603323 20632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:45.603626 20635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:45.604008 20633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:45.604562 20625 server_base.cc:1034] running on GCE node
I20260811 03:10:45.605814 20625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:45.607525 20625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:45.608817 20625 hybrid_clock.cc:648] HybridClock initialized: now 1786417845608748 us; error 38 us; skew 500 ppm
I20260811 03:10:45.609207 20625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:45 dist-test-slave-8tdl krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417845, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:45.612866 20625 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:10:45.614791 20625 webserver.cc:460] Webserver started at http://127.27.109.62:40209/ using document root <none> and password file <none>
I20260811 03:10:45.615593 20625 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:45.615739 20625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:45.616106 20625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:45.619617 20625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "3ceaee2b651e464ab3cb7d26b3167fa0"
format_stamp: "Formatted at 2026-08-11 03:10:45 on dist-test-slave-8tdl"
server_key: "6b686cbc386eb65dd930819d5bf59ad9"
server_key_iv: "716ed133bd3bdf90c611c76e58a56da3"
server_key_version: "encryptionkey@0"
I20260811 03:10:45.620532 20625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "3ceaee2b651e464ab3cb7d26b3167fa0"
format_stamp: "Formatted at 2026-08-11 03:10:45 on dist-test-slave-8tdl"
server_key: "6b686cbc386eb65dd930819d5bf59ad9"
server_key_iv: "716ed133bd3bdf90c611c76e58a56da3"
server_key_version: "encryptionkey@0"
I20260811 03:10:45.626544 20625 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260811 03:10:45.630555 20643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:45.632110 20625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 03:10:45.632329 20625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "3ceaee2b651e464ab3cb7d26b3167fa0"
format_stamp: "Formatted at 2026-08-11 03:10:45 on dist-test-slave-8tdl"
server_key: "6b686cbc386eb65dd930819d5bf59ad9"
server_key_iv: "716ed133bd3bdf90c611c76e58a56da3"
server_key_version: "encryptionkey@0"
I20260811 03:10:45.632618 20625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:45.654317 20625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:45.659217 20625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:45.659539 20625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:45.680651 20625 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:36491
I20260811 03:10:45.680696 20701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:36491 every 8 connection(s)
I20260811 03:10:45.682080 20625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:45.686192 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:45.690874 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 20625
I20260811 03:10:45.691116 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:45.691982 28084 external_mini_cluster.cc:1428] Setting key 4142469612449c77f31aabb771dfb0f3
I20260811 03:10:45.694696 20702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:45 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417845, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:10:45.720947 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.699014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.27.109.62:36491)
0811 03:10:45.700819 (+ 1805us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:45.700829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:45.700887 (+ 58us) server_negotiation.cc:407] Connection header received
0811 03:10:45.702537 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:45.702543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:45.702854 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:45.703078 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:45.704461 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:45.705672 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:45.707144 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:45.708086 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:45.708290 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:45.713058 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:45.713069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:45.713087 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:45.713128 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:45.716087 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:45.717080 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:45.717089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:45.717092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:45.717263 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:45.717781 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:45.717784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:45.717786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:45.718323 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:45.718487 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:45.719313 (+ 826us) server_negotiation.cc:299] Negotiation successful
0811 03:10:45.719613 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":1577,"thread_start_us":1336,"threads_started":1}
I20260811 03:10:45.720991 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.699013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36491 (local address 127.0.0.1:53134)
0811 03:10:45.699987 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.700036 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:45.702139 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:45.703097 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:45.703103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:45.703859 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:45.704210 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:45.704228 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:45.705841 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:45.705850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:45.707002 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:45.707015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:45.708355 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:45.708362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:45.708593 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:45.709332 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:45.709368 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:45.712772 (+ 3404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:45.716249 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:45.716265 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:45.716281 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:45.716919 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:45.717414 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:45.717422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:45.717429 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:45.717661 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:45.718478 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:45.718489 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:45.718871 (+ 382us) client_negotiation.cc:769] Sending connection context
0811 03:10:45.719176 (+ 305us) client_negotiation.cc:240] Negotiation successful
0811 03:10:45.719794 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":125,"thread_start_us":395,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.61@KRBTEST.COM; defaulting to no policy
I20260811 03:10:45.733769 20702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.61@KRBTEST.COM" created.
I20260811 03:10:45.735368 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.734807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51950)
0811 03:10:45.735096 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.735227 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260811 03:10:45.735677 20646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
I20260811 03:10:45.737484 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.737096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51952)
0811 03:10:45.737263 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.737335 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 03:10:45.737774 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:38547: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
I20260811 03:10:45.775309 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:38547
--webserver_interface=127.27.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36491,127.27.109.61:38547,127.27.109.60:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:10:45.787191 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } attempt: 1
I20260811 03:10:45.788712 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.788020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51956)
0811 03:10:45.788316 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.788493 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260811 03:10:45.790009 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.789585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51960)
0811 03:10:45.789785 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.789864 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260811 03:10:45.790313 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:38547: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
I20260811 03:10:45.858645 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } attempt: 2
I20260811 03:10:45.860144 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.859570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51974)
0811 03:10:45.859878 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.859986 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260811 03:10:45.861341 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.860933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51978)
0811 03:10:45.861101 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.861194 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260811 03:10:45.861626 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:38547: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
I20260811 03:10:45.952948 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } attempt: 3
I20260811 03:10:45.954195 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.953728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51982)
0811 03:10:45.953941 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.954043 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260811 03:10:45.955428 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:45.955061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51986)
0811 03:10:45.955214 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:45.955288 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 03:10:45.955626 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:38547: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
I20260811 03:10:46.098958 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } attempt: 4
I20260811 03:10:46.100426 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.099776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:51998)
0811 03:10:46.100137 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.100260 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260811 03:10:46.101667 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.101226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:52000)
0811 03:10:46.101397 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.101512 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260811 03:10:46.101886 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:38547: Network error: Client connection negotiation failed: client connection to 127.27.109.61:38547: connect: Connection refused (error 111)
W20260811 03:10:46.153393 20709 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 03:10:46.153824 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:46.153961 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:46.154109 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:46.161993 20709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:46.162148 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:46.162253 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:46.162356 20709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:46.162452 20709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:46.173092 20709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36491,127.27.109.61:38547,127.27.109.60:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:38547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:46.174590 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:46.176085 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:46.189615 20714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:46.189709 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:46.189711 20718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:46.190457 20709 server_base.cc:1034] running on GCE node
I20260811 03:10:46.191404 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:46.193024 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:46.194331 20709 hybrid_clock.cc:648] HybridClock initialized: now 1786417846194247 us; error 43 us; skew 500 ppm
I20260811 03:10:46.194818 20709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:46.198576 20709 init.cc:377] Logged in from keytab as kudu/127.27.109.61@KRBTEST.COM (short username kudu)
I20260811 03:10:46.200362 20709 webserver.cc:460] Webserver started at http://127.27.109.61:46545/ using document root <none> and password file <none>
I20260811 03:10:46.201164 20709 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:46.201292 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:46.201673 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:46.205250 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/data/instance:
uuid: "b426b3d2a0884032bf1c4fdbbe42272c"
format_stamp: "Formatted at 2026-08-11 03:10:46 on dist-test-slave-8tdl"
server_key: "94f17cb31f300562e998f7a1f8815ac6"
server_key_iv: "a12eee721c510dc45407b0a5df77b745"
server_key_version: "encryptionkey@0"
I20260811 03:10:46.206184 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/wal/instance:
uuid: "b426b3d2a0884032bf1c4fdbbe42272c"
format_stamp: "Formatted at 2026-08-11 03:10:46 on dist-test-slave-8tdl"
server_key: "94f17cb31f300562e998f7a1f8815ac6"
server_key_iv: "a12eee721c510dc45407b0a5df77b745"
server_key_version: "encryptionkey@0"
I20260811 03:10:46.212302 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260811 03:10:46.216460 20725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:46.218063 20709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:10:46.218276 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/wal
uuid: "b426b3d2a0884032bf1c4fdbbe42272c"
format_stamp: "Formatted at 2026-08-11 03:10:46 on dist-test-slave-8tdl"
server_key: "94f17cb31f300562e998f7a1f8815ac6"
server_key_iv: "a12eee721c510dc45407b0a5df77b745"
server_key_version: "encryptionkey@0"
I20260811 03:10:46.218534 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:46.255662 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:46.260578 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:46.260892 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:46.281587 20709 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:38547
I20260811 03:10:46.281618 20779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:38547 every 8 connection(s)
I20260811 03:10:46.283205 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
I20260811 03:10:46.284420 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 20709
I20260811 03:10:46.284637 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-1/wal/instance
I20260811 03:10:46.285257 28084 external_mini_cluster.cc:1428] Setting key bedb5699351a2f48c3b2dd8bd2ab70ec
I20260811 03:10:46.288132 20781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:46.295650 20781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.109.60@KRBTEST.COM; defaulting to no policy
I20260811 03:10:46.319201 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.298660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.27.109.62:36491)
0811 03:10:46.299322 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.299330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.300559 (+ 1229us) server_negotiation.cc:407] Connection header received
0811 03:10:46.302743 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.302749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.302860 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.302997 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.304831 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.305641 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.307489 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.307946 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.308128 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.312273 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.312279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.312282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.312326 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.315592 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.316566 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.316571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.316574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.316713 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.317192 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.317196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.317198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.317528 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.317699 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.318476 (+ 777us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.318769 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":340,"threads_started":1}
I20260811 03:10:46.320293 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.299400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36491 (local address 127.0.0.1:53142)
0811 03:10:46.300347 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.300411 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:46.302354 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:46.303298 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:46.303307 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:46.303923 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:46.304567 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.304588 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.305845 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.305849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.307310 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.307325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.308247 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.308255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.308508 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.309304 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:46.309331 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:46.311971 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:46.315809 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.315820 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.315849 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.316388 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.316840 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.316846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.316849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.317050 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.317672 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:46.317681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:46.318089 (+ 408us) client_negotiation.cc:769] Sending connection context
0811 03:10:46.318388 (+ 299us) client_negotiation.cc:240] Negotiation successful
0811 03:10:46.319001 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":350,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.60@KRBTEST.COM" created.
I20260811 03:10:46.322250 20781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:10:46.341902 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.323544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.27.109.61:38547)
0811 03:10:46.324364 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.324375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.324413 (+ 38us) server_negotiation.cc:407] Connection header received
0811 03:10:46.324482 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.324494 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.324853 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.325031 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.325954 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.327817 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.329097 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.330099 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.330251 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.333814 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.333835 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.333860 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.333916 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.338177 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.339156 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.339169 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.339175 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.339348 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.339873 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.339877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.339879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.340459 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.340716 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.341219 (+ 503us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.341571 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":584,"thread_start_us":419,"threads_started":1}
I20260811 03:10:46.341938 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.323402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:52002)
0811 03:10:46.323656 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.323678 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:46.323815 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:46.325145 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:46.325150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:46.325594 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:46.325755 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.325764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.328023 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.328027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.328937 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.328949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.330257 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.330261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.330406 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.331108 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:46.331127 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:46.333548 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:46.338380 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.338385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.338388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.338992 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.339522 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.339531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.339535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.339722 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.340641 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:46.340648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:46.340856 (+ 208us) client_negotiation.cc:769] Sending connection context
0811 03:10:46.341048 (+ 192us) client_negotiation.cc:240] Negotiation successful
0811 03:10:46.341422 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 03:10:46.344937 20781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:10:46.346421 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.345870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55772)
0811 03:10:46.346102 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.346224 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260811 03:10:46.346817 20729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.348591 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.348199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55774)
0811 03:10:46.348375 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.348463 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260811 03:10:46.348836 20781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
I20260811 03:10:46.371433 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:46167
--webserver_interface=127.27.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:36491,127.27.109.61:38547,127.27.109.60:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:10:46.398197 20781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 1
I20260811 03:10:46.399493 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.398979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55788)
0811 03:10:46.399214 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.399331 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260811 03:10:46.400803 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.400282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55794)
0811 03:10:46.400477 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.400627 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260811 03:10:46.401098 20781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.401144 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } attempt: 5
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417845, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:10:46.417274 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.401802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:52016)
0811 03:10:46.402030 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.402054 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:46.402217 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:46.402589 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:46.402598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:46.403229 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:46.403525 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.403543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.404581 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.404590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.406297 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.406320 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.407288 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.407297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.407623 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.408390 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:46.408419 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:46.411135 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:46.414499 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.414505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.414509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.415032 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.415439 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.415444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.415448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.415645 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.416224 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:46.416229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:46.416404 (+ 175us) client_negotiation.cc:769] Sending connection context
0811 03:10:46.416546 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 03:10:46.416911 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 03:10:46.417272 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.401879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.27.109.61:38547)
0811 03:10:46.402059 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.402065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.402133 (+ 68us) server_negotiation.cc:407] Connection header received
0811 03:10:46.402353 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.402357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.402418 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.402502 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.403723 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.404415 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.406513 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.407096 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.407213 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.411377 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.411382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.411386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.411423 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.414318 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.415170 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.415182 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.415186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.415314 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.415784 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.415788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.415790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.416089 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.416258 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.416689 (+ 431us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.416928 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 03:10:46.418361 20702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:10:46.419605 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.419146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55808)
0811 03:10:46.419357 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.419449 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260811 03:10:46.420801 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.420457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55810)
0811 03:10:46.420607 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.420698 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260811 03:10:46.421169 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.469389 20781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 2
I20260811 03:10:46.470850 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.470168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55826)
0811 03:10:46.470451 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.470585 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260811 03:10:46.472095 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.471666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55834)
0811 03:10:46.471842 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.471938 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260811 03:10:46.472343 20781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.472401 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 1
I20260811 03:10:46.473461 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.473047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55850)
0811 03:10:46.473223 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.473308 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 03:10:46.474581 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.474224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55862)
0811 03:10:46.474383 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.474460 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 03:10:46.475240 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.543495 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 2
I20260811 03:10:46.544754 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.544232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55872)
0811 03:10:46.544487 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.544582 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260811 03:10:46.545769 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.545439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55876)
0811 03:10:46.545594 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.545667 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 03:10:46.545975 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.563671 20781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 3
I20260811 03:10:46.565073 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.564451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55878)
0811 03:10:46.564728 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.564849 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260811 03:10:46.566643 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.566084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55890)
0811 03:10:46.566361 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.566472 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260811 03:10:46.566996 20781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.652217 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 3
I20260811 03:10:46.653451 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.652928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55906)
0811 03:10:46.653146 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.653243 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260811 03:10:46.654726 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.654230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55916)
0811 03:10:46.654482 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.654561 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260811 03:10:46.655007 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.710278 20781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 4
I20260811 03:10:46.711781 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.711155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55922)
0811 03:10:46.711451 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.711577 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260811 03:10:46.712982 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.712586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55930)
0811 03:10:46.712752 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.712845 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 03:10:46.713284 20781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
W20260811 03:10:46.758415 20790 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 03:10:46.758874 20790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:46.758984 20790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:46.759130 20790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:46.766939 20790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:46.767045 20790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:46.767139 20790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:46.767213 20790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:46.767292 20790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:46.777709 20790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:36491,127.27.109.61:38547,127.27.109.60:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:46167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:46.779279 20790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:46.780735 20790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:46.793876 20799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:46.794034 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:46.793907 20795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:46.794835 20790 server_base.cc:1034] running on GCE node
I20260811 03:10:46.795708 20790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:46.797385 20790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:46.798655 20790 hybrid_clock.cc:648] HybridClock initialized: now 1786417846798580 us; error 37 us; skew 500 ppm
I20260811 03:10:46.799176 20790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:46.803088 20790 init.cc:377] Logged in from keytab as kudu/127.27.109.60@KRBTEST.COM (short username kudu)
I20260811 03:10:46.804940 20790 webserver.cc:460] Webserver started at http://127.27.109.60:46153/ using document root <none> and password file <none>
I20260811 03:10:46.805773 20790 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:46.805931 20790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:46.806319 20790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:46.811029 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/data/instance:
uuid: "30fd4b0e2b654094b3335c999bb049e3"
format_stamp: "Formatted at 2026-08-11 03:10:46 on dist-test-slave-8tdl"
server_key: "1c4211315e7f2ade245d8101698afe9c"
server_key_iv: "82e9a01bc8018d660b54e334b3804d74"
server_key_version: "encryptionkey@0"
I20260811 03:10:46.812090 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/wal/instance:
uuid: "30fd4b0e2b654094b3335c999bb049e3"
format_stamp: "Formatted at 2026-08-11 03:10:46 on dist-test-slave-8tdl"
server_key: "1c4211315e7f2ade245d8101698afe9c"
server_key_iv: "82e9a01bc8018d660b54e334b3804d74"
server_key_version: "encryptionkey@0"
I20260811 03:10:46.818815 20790 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260811 03:10:46.823290 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:46.824939 20790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260811 03:10:46.825170 20790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/wal
uuid: "30fd4b0e2b654094b3335c999bb049e3"
format_stamp: "Formatted at 2026-08-11 03:10:46 on dist-test-slave-8tdl"
server_key: "1c4211315e7f2ade245d8101698afe9c"
server_key_iv: "82e9a01bc8018d660b54e334b3804d74"
server_key_version: "encryptionkey@0"
I20260811 03:10:46.825496 20790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:46.832262 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 4
I20260811 03:10:46.833453 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.832966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55942)
0811 03:10:46.833226 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.833314 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 03:10:46.834508 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.834147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55956)
0811 03:10:46.834299 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.834376 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 03:10:46.834761 20702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:46167: Network error: Client connection negotiation failed: client connection to 127.27.109.60:46167: connect: Connection refused (error 111)
I20260811 03:10:46.848644 20790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:46.853606 20790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:46.853922 20790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:46.874562 20790 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:46167
I20260811 03:10:46.874605 20859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:46167 every 8 connection(s)
I20260811 03:10:46.876111 20790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
I20260811 03:10:46.880055 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 20790
I20260811 03:10:46.880267 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786417454123933-28084-0/minicluster-data/master-2/wal/instance
I20260811 03:10:46.880522 20860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:46.880883 28084 external_mini_cluster.cc:1428] Setting key 36683b1b745500f40e77ab2b43a0d4b6
I20260811 03:10:46.891521 20860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:10:46.907060 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.882973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.27.109.60:46167)
0811 03:10:46.883823 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.883836 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.883880 (+ 44us) server_negotiation.cc:407] Connection header received
0811 03:10:46.885719 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.885744 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.886183 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.886553 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.888727 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.890139 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.892026 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.892840 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.892990 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.898046 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.898061 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.898072 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.898114 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.901298 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.902501 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.902508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.902515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.902762 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.903261 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.903269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.903272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.904099 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.904354 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.904956 (+ 602us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.905458 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":382,"threads_started":1}
I20260811 03:10:46.912091 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.894867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:36491 (local address 127.0.0.1:53154)
0811 03:10:46.895930 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.895988 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:46.896175 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:46.896640 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:46.896646 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:46.897464 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:46.897725 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.897745 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.898865 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.898869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.900048 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.900062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.900891 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.900898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.901079 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.901726 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:46.901762 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:46.904738 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:46.908686 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.908699 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.908708 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.909347 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.909856 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.909861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.909864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.910018 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.910809 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:46.910817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:46.911076 (+ 259us) client_negotiation.cc:769] Sending connection context
0811 03:10:46.911324 (+ 248us) client_negotiation.cc:240] Negotiation successful
0811 03:10:46.911600 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":611,"threads_started":1}
I20260811 03:10:46.913530 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.894899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.27.109.62:36491)
0811 03:10:46.895537 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.895543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.896102 (+ 559us) server_negotiation.cc:407] Connection header received
0811 03:10:46.896345 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.896350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.896437 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.896584 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.897900 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.898630 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.900222 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.900664 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.900815 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.904977 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.904984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.904990 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.905033 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.908540 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.909519 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.909528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.909533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.909717 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.910154 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.910160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.910163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.910565 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.912441 (+ 1876us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.912644 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.913051 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":271,"threads_started":1}
I20260811 03:10:46.915283 20860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:10:46.930296 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.910801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.27.109.62:36491)
0811 03:10:46.911465 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.911469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.911488 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:10:46.911527 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.911531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.911598 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.911683 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.913763 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.915090 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.917039 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.917771 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.917921 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.923415 (+ 5494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.923421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.923425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.923462 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.927311 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.928094 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.928101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.928105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.928269 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.928690 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.928696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.928699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.928996 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.929300 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.929626 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.929859 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":339,"threads_started":1}
I20260811 03:10:46.931887 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.916139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:38547 (local address 127.0.0.1:52026)
0811 03:10:46.916324 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.916346 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:46.916457 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:46.916888 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:46.916891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:46.917503 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:46.917757 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.917768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.919403 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.919408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.920326 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.920338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.921511 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.921518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.921734 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.922489 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:46.922510 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:46.924938 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:46.928640 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.928646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.928650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.929065 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.929465 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.929512 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.929515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.929675 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.930889 (+ 1214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:46.930899 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:46.931047 (+ 148us) client_negotiation.cc:769] Sending connection context
0811 03:10:46.931211 (+ 164us) client_negotiation.cc:240] Negotiation successful
0811 03:10:46.931528 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 03:10:46.931915 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.916248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.27.109.61:38547)
0811 03:10:46.916432 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.916454 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.916472 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:10:46.916602 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.916607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.916687 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.916804 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.918015 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.919227 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.920497 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.921304 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.921472 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.925166 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.925175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.925179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.925231 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.928494 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.929192 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.929198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.929202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.929340 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.929788 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.929793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.929796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.930290 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.930443 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.931304 (+ 861us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.931528 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 03:10:46.933297 20860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
Aug 11 03:10:46 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260811 03:10:46.948110 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.932493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.27.109.61:38547)
0811 03:10:46.932666 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.932669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.932683 (+ 14us) server_negotiation.cc:407] Connection header received
0811 03:10:46.932722 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.932725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.932807 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.932903 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.934506 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.935318 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.936499 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.936965 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.937106 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.942218 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.942223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.942226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.942262 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.945263 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.945977 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.945983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.945989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.946132 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.946528 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.946534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.946539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.946885 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.947173 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.947473 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.947725 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 03:10:46.948531 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.934262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.27.109.60:46167)
0811 03:10:46.934417 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:46.934423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:46.934453 (+ 30us) server_negotiation.cc:407] Connection header received
0811 03:10:46.934500 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:46.934504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:46.934574 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:46.934671 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:46.935410 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.936140 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.937473 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.939115 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.939255 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.942561 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:46.942568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:46.942575 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:46.942641 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:46.945495 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.946157 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.946164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.946167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.946289 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:46.946649 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:46.946656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:46.946659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:46.946988 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:46.947144 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:46.947532 (+ 388us) server_negotiation.cc:299] Negotiation successful
0811 03:10:46.947965 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":65}
I20260811 03:10:46.948180 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:46.934040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55978)
0811 03:10:46.934241 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:46.934261 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:46.934395 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:46.934722 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:46.934726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:46.935136 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:46.935265 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.935273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.936265 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.936272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.937325 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:46.937337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:46.939297 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:46.939304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:46.939525 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:46.940097 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:46.940115 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:46.942342 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:46.945652 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.945659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.945663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.946037 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.946404 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:46.946412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:46.946417 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:46.946544 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:46.947104 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:46.947109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:46.947257 (+ 148us) client_negotiation.cc:769] Sending connection context
0811 03:10:46.947407 (+ 150us) client_negotiation.cc:240] Negotiation successful
0811 03:10:46.947767 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 03:10:46.953667 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:46.954574 20860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: Bootstrap starting.
I20260811 03:10:46.958474 20860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:46.959753 20860 log.cc:824] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:46.957336 20672 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53160
I20260811 03:10:46.963268 20860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: No bootstrap required, opened a new log
I20260811 03:10:46.971568 20860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } }
I20260811 03:10:46.972066 20860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fd4b0e2b654094b3335c999bb049e3, State: Initialized, Role: FOLLOWER
I20260811 03:10:46.972584 20860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } }
I20260811 03:10:46.973717 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:46.974159 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:46.974948 20860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 03:10:46.985913 20882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:10:46.986078 20882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:10:46.983805 20790 master_runner.cc:186] Error getting master registration for 127.27.109.62:36491: OK, instance_id { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" instance_seqno: 1786417845674756 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 03:10:46.997443 20790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 03:10:47.012609 20781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 5
Aug 11 03:10:47 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417846, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260811 03:10:47.032797 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:47.013628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:55994)
0811 03:10:47.013844 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:47.013868 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:47.014016 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:47.014614 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:47.014620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:47.015312 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:47.015757 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:47.015778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.017597 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.017692 (+ 95us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:47.019313 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:47.019332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.020687 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.020695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:47.021092 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:47.021863 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:47.021883 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:47.024675 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:47.029651 (+ 4976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:47.029657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:47.029662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:47.030308 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:47.030818 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:47.030823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:47.030827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:47.031013 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:47.031687 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:47.031692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:47.031872 (+ 180us) client_negotiation.cc:769] Sending connection context
0811 03:10:47.032042 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 03:10:47.032403 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 03:10:47.032810 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:47.013830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.27.109.60:46167)
0811 03:10:47.014105 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.014112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.014141 (+ 29us) server_negotiation.cc:407] Connection header received
0811 03:10:47.014220 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.014227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.014327 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.014486 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.015970 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.017401 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.019623 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.020468 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.020663 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:47.024947 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:47.024958 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:47.024964 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:47.025016 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:47.029415 (+ 4399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.030502 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.030507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.030509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.030649 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.031150 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.031155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.031158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.031549 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:47.031739 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:47.032168 (+ 429us) server_negotiation.cc:299] Negotiation successful
0811 03:10:47.032408 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260811 03:10:47.038169 20781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: Bootstrap starting.
I20260811 03:10:47.041728 20781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:47.043212 20781 log.cc:824] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:47.046187 20781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: No bootstrap required, opened a new log
I20260811 03:10:47.057863 20781 raft_consensus.cc:348] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } }
I20260811 03:10:47.058327 20781 raft_consensus.cc:729] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b426b3d2a0884032bf1c4fdbbe42272c, State: Initialized, Role: FOLLOWER
I20260811 03:10:47.058919 20781 consensus_queue.cc:260] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } }
I20260811 03:10:47.060030 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.060447 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:47.061326 20781 sys_catalog.cc:564] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [sys.catalog]: configured and running, proceeding with master startup.
W20260811 03:10:47.072142 20896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:10:47.072346 20896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:10:47.069533 20709 master_runner.cc:186] Error getting master registration for 127.27.109.62:36491: OK, instance_id { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" instance_seqno: 1786417845674756 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 03:10:47.083415 20709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 03:10:47.106617 20812 tablet.cc:2369] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:10:47.112057 20702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } attempt: 5
Aug 11 03:10:47 dist-test-slave-8tdl krb5kdc[20611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417845, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260811 03:10:47.131982 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:47.112920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:46167 (local address 127.0.0.1:56006)
0811 03:10:47.113398 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:47.113421 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:47.113592 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:47.114024 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:47.114030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:47.114633 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:47.114926 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:47.114941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.116437 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.116443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:47.118130 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:47.118147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.119262 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.119270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:47.119500 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:47.120300 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:47.120329 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:47.123565 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:47.128393 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:47.128401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:47.128406 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:47.129069 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:47.129581 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:47.129586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:47.129590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:47.129807 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:47.130771 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:47.130777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:47.130986 (+ 209us) client_negotiation.cc:769] Sending connection context
0811 03:10:47.131180 (+ 194us) client_negotiation.cc:240] Negotiation successful
0811 03:10:47.131532 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317}
I20260811 03:10:47.132412 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:47.113137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.27.109.60:46167)
0811 03:10:47.113398 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.113405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.113494 (+ 89us) server_negotiation.cc:407] Connection header received
0811 03:10:47.113758 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.113764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.113857 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.113991 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.115133 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.116296 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.118332 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.119102 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.119236 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:47.123894 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:47.123902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:47.123908 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:47.123966 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:47.128194 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.129222 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.129229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.129233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.129428 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.129981 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.129993 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.129997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.130537 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:47.130769 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:47.131324 (+ 555us) server_negotiation.cc:299] Negotiation successful
0811 03:10:47.131652 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260811 03:10:47.139822 20702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0: Bootstrap starting.
I20260811 03:10:47.145033 20702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:47.146811 20702 log.cc:824] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:47.151098 20702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0: No bootstrap required, opened a new log
I20260811 03:10:47.160434 20702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } }
I20260811 03:10:47.161074 20702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ceaee2b651e464ab3cb7d26b3167fa0, State: Initialized, Role: FOLLOWER
I20260811 03:10:47.161791 20702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } }
I20260811 03:10:47.163444 20899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.164316 20899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:47.165474 20702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:47.165568 20672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 03:10:47.165791 20672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:47.169437 20672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } }
I20260811 03:10:47.171998 20752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b426b3d2a0884032bf1c4fdbbe42272c"
I20260811 03:10:47.172562 20752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:47.176803 20752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ceaee2b651e464ab3cb7d26b3167fa0 in term 1.
I20260811 03:10:47.177827 20646 leader_election.cc:302] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ceaee2b651e464ab3cb7d26b3167fa0, b426b3d2a0884032bf1c4fdbbe42272c; no voters:
I20260811 03:10:47.179132 20672 leader_election.cc:288] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [CANDIDATE]: Term 1 election: Requested vote from peers b426b3d2a0884032bf1c4fdbbe42272c (127.27.109.61:38547), 30fd4b0e2b654094b3335c999bb049e3 (127.27.109.60:46167)
I20260811 03:10:47.180001 20899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:47.181866 20833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "30fd4b0e2b654094b3335c999bb049e3"
I20260811 03:10:47.182482 20833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:47.187489 20899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 3ceaee2b651e464ab3cb7d26b3167fa0, State: Running, Role: LEADER
I20260811 03:10:47.187842 20833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ceaee2b651e464ab3cb7d26b3167fa0 in term 1.
I20260811 03:10:47.188721 20899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } }
I20260811 03:10:47.198359 20902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ceaee2b651e464ab3cb7d26b3167fa0. Latest consensus state: current_term: 1 leader_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.199034 20902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:47.204609 20905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 03:10:47.225612 20905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260811 03:10:47.263991 20731 tablet.cc:2369] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:10:47.265465 20833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [term 1 FOLLOWER]: Refusing update from remote peer 3ceaee2b651e464ab3cb7d26b3167fa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:10:47.266955 20752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [term 1 FOLLOWER]: Refusing update from remote peer 3ceaee2b651e464ab3cb7d26b3167fa0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:10:47.268220 20899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:10:47.269920 20899 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:10:47.273123 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:47.214378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.27.109.61:38547)
0811 03:10:47.214605 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.214614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.223276 (+ 8662us) server_negotiation.cc:407] Connection header received
0811 03:10:47.223635 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.223642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.223741 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.223824 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.229781 (+ 5957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.230955 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.243272 (+ 12317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.244034 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.244155 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:47.253627 (+ 9472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:47.253636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:47.253641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:47.253693 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:47.257560 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.265150 (+ 7590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.265158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.265164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.265356 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.266020 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.266026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.266028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.266477 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:47.266794 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:47.272377 (+ 5583us) server_negotiation.cc:299] Negotiation successful
0811 03:10:47.272708 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 03:10:47.275074 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:47.215690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.27.109.62:36491)
0811 03:10:47.215915 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.215922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.226887 (+ 10965us) server_negotiation.cc:407] Connection header received
0811 03:10:47.227174 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.227179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.227292 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.227366 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.239304 (+ 11938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.240879 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.249005 (+ 8126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.249794 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.249914 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:47.262249 (+ 12335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:47.262256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:47.262261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:47.262307 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:47.269480 (+ 7173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.270387 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.270393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.270399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.270564 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.271081 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.271086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.271089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.271510 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:47.271768 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:47.274382 (+ 2614us) server_negotiation.cc:299] Negotiation successful
0811 03:10:47.274724 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 03:10:47.278326 20927 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 03:10:47.292346 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:47.227641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.27.109.60:46167)
0811 03:10:47.227892 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.227901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.235469 (+ 7568us) server_negotiation.cc:407] Connection header received
0811 03:10:47.242908 (+ 7439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.242918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.243103 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.243221 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.246937 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.248573 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.251597 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:47.252675 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:47.252819 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:47.262341 (+ 9522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:47.262352 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:47.262360 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:47.262424 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:47.272174 (+ 9750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.273454 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.273465 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.273472 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.273688 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:47.289045 (+ 15357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:47.289056 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:47.289062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:47.289654 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:47.289972 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:47.291560 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0811 03:10:47.291868 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260811 03:10:47.311327 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ceaee2b651e464ab3cb7d26b3167fa0. Latest consensus state: current_term: 1 leader_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.311704 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:47.322552 20928 mvcc.cc:205] Tried to move back new op lower bound from 7317167502381629440 to 7317167502092001280. Current Snapshot: MvccSnapshot[applied={T|T < 7317167502381629440}]
I20260811 03:10:47.342636 20902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.343058 20902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:47.348711 20905 catalog_manager.cc:1347] Generated new cluster ID: eb60004175d542a18dbd0dd1a06d4451
I20260811 03:10:47.348866 20905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:47.362780 20885 sys_catalog.cc:455] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.363132 20885 sys_catalog.cc:458] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:47.376629 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ceaee2b651e464ab3cb7d26b3167fa0. Latest consensus state: current_term: 1 leader_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.377015 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:47.383576 20899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.383941 20899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:47.385792 20929 mvcc.cc:205] Tried to move back new op lower bound from 7317167502381629440 to 7317167502092001280. Current Snapshot: MvccSnapshot[applied={T|T < 7317167502381629440}]
I20260811 03:10:47.397917 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ceaee2b651e464ab3cb7d26b3167fa0" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 36491 } } peers { permanent_uuid: "b426b3d2a0884032bf1c4fdbbe42272c" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 38547 } } peers { permanent_uuid: "30fd4b0e2b654094b3335c999bb049e3" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 46167 } } }
I20260811 03:10:47.398276 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:47.404570 20905 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:47.406565 20905 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:47.443424 20905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ceaee2b651e464ab3cb7d26b3167fa0: Generated new TSK 0
I20260811 03:10:47.444689 20905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 03:10:47.600893 20867 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.589901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.27.109.62:36491)
0811 03:10:47.590146 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.590154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.596492 (+ 6338us) server_negotiation.cc:407] Connection header received
0811 03:10:47.598355 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.598385 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.598507 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.598606 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.600534 (+ 1928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53178: BlockingRecv error: recv got EOF from 127.0.0.1:53178 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260811 03:10:47.602931 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.583056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.27.109.61:38547)
0811 03:10:47.583334 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.583342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.595861 (+ 12519us) server_negotiation.cc:407] Connection header received
0811 03:10:47.598921 (+ 3060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.598928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.599024 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.599118 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.602614 (+ 3496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52054: BlockingRecv error: recv got EOF from 127.0.0.1:52054 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260811 03:10:47.606292 20863 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.597202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.27.109.60:46167)
0811 03:10:47.597407 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.597414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.604193 (+ 6779us) server_negotiation.cc:407] Connection header received
0811 03:10:47.604616 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.604622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.604725 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.604811 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.605993 (+ 1182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56022: BlockingRecv error: recv got EOF from 127.0.0.1:56022 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 03:10:47.652175 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.638469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.27.109.61:38547)
0811 03:10:47.638697 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.638705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.649513 (+ 10808us) server_negotiation.cc:407] Connection header received
0811 03:10:47.650154 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.650162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.650338 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.650445 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.651817 (+ 1372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52068: BlockingRecv error: recv got EOF from 127.0.0.1:52068 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260811 03:10:47.713207 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.705156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52078 (local address 127.27.109.61:38547)
0811 03:10:47.705381 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.705390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.711491 (+ 6101us) server_negotiation.cc:407] Connection header received
0811 03:10:47.711695 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.711703 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.711812 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.711951 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.712945 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52078: BlockingRecv error: recv got EOF from 127.0.0.1:52078 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260811 03:10:47.728034 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.719041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.27.109.61:38547)
0811 03:10:47.719236 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.719243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.719624 (+ 381us) server_negotiation.cc:407] Connection header received
0811 03:10:47.719835 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.719841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.719968 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.720088 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.727747 (+ 7659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52086: BlockingRecv error: recv got EOF from 127.0.0.1:52086 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260811 03:10:47.740979 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.736805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.27.109.61:38547)
0811 03:10:47.737058 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.737064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.739337 (+ 2273us) server_negotiation.cc:407] Connection header received
0811 03:10:47.739544 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.739551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.739657 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.739784 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.740693 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52102: BlockingRecv error: recv got EOF from 127.0.0.1:52102 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260811 03:10:47.755957 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.753697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.27.109.61:38547)
0811 03:10:47.753936 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.753943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.754244 (+ 301us) server_negotiation.cc:407] Connection header received
0811 03:10:47.754429 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.754435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.754549 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.754721 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.755633 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52106: BlockingRecv error: recv got EOF from 127.0.0.1:52106 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260811 03:10:47.778038 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.775527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.27.109.61:38547)
0811 03:10:47.775727 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.775731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.776415 (+ 684us) server_negotiation.cc:407] Connection header received
0811 03:10:47.776683 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.776696 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.776784 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.776899 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.777763 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52112: BlockingRecv error: recv got EOF from 127.0.0.1:52112 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260811 03:10:47.817494 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.815086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.27.109.61:38547)
0811 03:10:47.815375 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.815385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.815821 (+ 436us) server_negotiation.cc:407] Connection header received
0811 03:10:47.816008 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.816015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.816140 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.816277 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.817142 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52128: BlockingRecv error: recv got EOF from 127.0.0.1:52128 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260811 03:10:47.887948 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:47.885238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.27.109.61:38547)
0811 03:10:47.885565 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:47.885575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:47.886144 (+ 569us) server_negotiation.cc:407] Connection header received
0811 03:10:47.886395 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:47.886399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:47.886518 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:47.886639 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:47.887729 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52144: BlockingRecv error: recv got EOF from 127.0.0.1:52144 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
I20260811 03:10:47.988775 20882 catalog_manager.cc:1259] Loaded cluster ID: eb60004175d542a18dbd0dd1a06d4451
I20260811 03:10:47.988896 20882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: loading cluster ID for follower catalog manager: success
I20260811 03:10:47.992343 20882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: acquiring CA information for follower catalog manager: success
I20260811 03:10:47.994472 20882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30fd4b0e2b654094b3335c999bb049e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 03:10:48.023033 20787 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:48.020593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.27.109.61:38547)
0811 03:10:48.020921 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:48.020929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:48.021255 (+ 326us) server_negotiation.cc:407] Connection header received
0811 03:10:48.021534 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:48.021542 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:48.021640 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:48.021765 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:48.022715 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52146: BlockingRecv error: recv got EOF from 127.0.0.1:52146 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260811 03:10:48.074433 20896 catalog_manager.cc:1259] Loaded cluster ID: eb60004175d542a18dbd0dd1a06d4451
I20260811 03:10:48.074539 20896 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: loading cluster ID for follower catalog manager: success
I20260811 03:10:48.077212 20896 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: acquiring CA information for follower catalog manager: success
I20260811 03:10:48.079149 20896 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b426b3d2a0884032bf1c4fdbbe42272c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 03:10:48.292105 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:48.284543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.27.109.61:38547)
0811 03:10:48.284797 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:48.284804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:48.285200 (+ 396us) server_negotiation.cc:407] Connection header received
0811 03:10:48.285407 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:48.285411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:48.285604 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:48.285707 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:48.286377 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.287296 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.289533 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.290001 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.290131 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:48.290808 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:48.291369 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:48.291490 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:48.291679 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 03:10:48.291753 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 03:10:48.307056 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:48.300814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.27.109.60:46167)
0811 03:10:48.301613 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:48.301618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:48.301637 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:10:48.301738 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:48.301742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:48.301909 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:48.302091 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:48.302550 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.303408 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.304825 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.305284 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.305393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:48.305909 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:48.306457 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:48.306608 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:48.306656 (+ 48us) server_negotiation.cc:299] Negotiation successful
0811 03:10:48.306815 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":547,"threads_started":1}
I20260811 03:10:48.326506 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:48.319249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.27.109.61:38547)
0811 03:10:48.319511 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:48.319521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:48.320190 (+ 669us) server_negotiation.cc:407] Connection header received
0811 03:10:48.320507 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:48.320513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:48.320613 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:48.320687 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:48.321393 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.322564 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.324465 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.324911 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.325010 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:48.325572 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:48.325690 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:48.325841 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:48.326009 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 03:10:48.326149 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 03:10:48.328996 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:48.319126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.27.109.62:36491)
0811 03:10:48.319626 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:48.319633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:48.320493 (+ 860us) server_negotiation.cc:407] Connection header received
0811 03:10:48.320945 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:48.320954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:48.321252 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:48.321446 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:48.322066 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.323256 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.326613 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.327215 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.327366 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:48.327911 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:48.328287 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:48.328373 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:48.328577 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 03:10:48.328671 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":254,"threads_started":1}
I20260811 03:10:48.329520 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:48.320689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.27.109.60:46167)
0811 03:10:48.320881 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:48.320887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:48.322549 (+ 1662us) server_negotiation.cc:407] Connection header received
0811 03:10:48.322742 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:48.322747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:48.322839 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:48.322932 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:48.323568 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.324266 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.327031 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:48.327645 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:48.327798 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:48.328838 (+ 1040us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:48.328944 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:48.329042 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:48.329172 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 03:10:48.329257 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 03:10:48.335114 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20625
I20260811 03:10:48.336508 20697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:48.454838 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 20625
I20260811 03:10:48.481813 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20709
I20260811 03:10:48.483072 20775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:48.597916 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 20709
I20260811 03:10:48.623930 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 20790
I20260811 03:10:48.625293 20855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:48.737135 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 20790
2026-08-11T03:10:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5174 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 03:10:48 dist-test-slave-8tdl krb5kdc[21019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 03:10:48 dist-test-slave-8tdl krb5kdc[21019](info): set up 2 sockets
Aug 11 03:10:48 dist-test-slave-8tdl krb5kdc[21019](info): commencing operation
krb5kdc: starting...
W20260811 03:10:49.978578 28084 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.004s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/test-user.keytab.
Aug 11 03:10:50 dist-test-slave-8tdl krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T03:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T03:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.109.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
Entry for principal kudu/127.27.109.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab.
I20260811 03:10:50.359493 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38103
--webserver_interface=127.27.109.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38103,127.27.109.61:39563,127.27.109.60:37303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 03:10:50.729416 21033 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 03:10:50.729846 21033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:50.729993 21033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:50.730144 21033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:50.737848 21033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:50.737972 21033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:50.738034 21033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:50.738127 21033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:50.738233 21033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:50.749075 21033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38103,127.27.109.61:39563,127.27.109.60:37303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.62.keytab
--principal=kudu/127.27.109.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.62:38103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.109.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:50.750711 21033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:50.752198 21033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:50.765496 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:50.765863 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:50.766050 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:50.767349 21033 server_base.cc:1034] running on GCE node
I20260811 03:10:50.768388 21033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:50.770046 21033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:50.771328 21033 hybrid_clock.cc:648] HybridClock initialized: now 1786417850771255 us; error 46 us; skew 500 ppm
I20260811 03:10:50.771733 21033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:50 dist-test-slave-8tdl krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417850, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:50.779304 21033 init.cc:377] Logged in from keytab as kudu/127.27.109.62@KRBTEST.COM (short username kudu)
I20260811 03:10:50.781131 21033 webserver.cc:460] Webserver started at http://127.27.109.62:37757/ using document root <none> and password file <none>
I20260811 03:10:50.781924 21033 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:50.782073 21033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:50.782421 21033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:50.785851 21033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/data/instance:
uuid: "dcce9e26f9234ebaa793367eb2aab8f8"
format_stamp: "Formatted at 2026-08-11 03:10:50 on dist-test-slave-8tdl"
server_key: "b10d323aa2707605666411d164084520"
server_key_iv: "66d443d9dc22a55073792f1e01029814"
server_key_version: "encryptionkey@0"
I20260811 03:10:50.787554 21033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/wal/instance:
uuid: "dcce9e26f9234ebaa793367eb2aab8f8"
format_stamp: "Formatted at 2026-08-11 03:10:50 on dist-test-slave-8tdl"
server_key: "b10d323aa2707605666411d164084520"
server_key_iv: "66d443d9dc22a55073792f1e01029814"
server_key_version: "encryptionkey@0"
I20260811 03:10:50.793641 21033 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 03:10:50.797896 21050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:50.799575 21033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 03:10:50.799810 21033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/wal
uuid: "dcce9e26f9234ebaa793367eb2aab8f8"
format_stamp: "Formatted at 2026-08-11 03:10:50 on dist-test-slave-8tdl"
server_key: "b10d323aa2707605666411d164084520"
server_key_iv: "66d443d9dc22a55073792f1e01029814"
server_key_version: "encryptionkey@0"
I20260811 03:10:50.800122 21033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:50.814602 21033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:50.819676 21033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:50.819993 21033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:50.841501 21033 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.62:38103
I20260811 03:10:50.841470 21105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.62:38103 every 8 connection(s)
I20260811 03:10:50.843184 21033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/data/info.pb
I20260811 03:10:50.847208 21106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:50.848764 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 21033
I20260811 03:10:50.848973 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-0/wal/instance
I20260811 03:10:50.849540 28084 external_mini_cluster.cc:1428] Setting key 9b271810885a5c2f4c4e3bfb4e226f0a
I20260811 03:10:50.857712 21106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:50 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417850, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:10:50.886921 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:50.862445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.27.109.62:38103)
0811 03:10:50.864608 (+ 2163us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:50.864615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:50.864663 (+ 48us) server_negotiation.cc:407] Connection header received
0811 03:10:50.867085 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:50.867098 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:50.867543 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:50.867804 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:50.869163 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:50.871248 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:50.872764 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:50.873829 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:50.874046 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:50.878400 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:50.878411 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:50.878425 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:50.878459 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:50.881757 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:50.882823 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:50.882830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:50.882833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:50.883000 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:50.883628 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:50.883632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:50.883637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:50.884376 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:50.884620 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:50.885392 (+ 772us) server_negotiation.cc:299] Negotiation successful
0811 03:10:50.885699 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":1947,"spinlock_wait_cycles":90880,"thread_start_us":1657,"threads_started":1}
I20260811 03:10:50.886965 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:50.862445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38103 (local address 127.0.0.1:49540)
0811 03:10:50.863527 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:50.863584 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:50.866533 (+ 2949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:50.867853 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:50.867858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:50.868526 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:50.868905 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:50.868924 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:50.871425 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:50.871429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:50.872599 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:50.872613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:50.874099 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:50.874106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:50.874274 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:50.875094 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:50.875119 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:50.877902 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:50.881929 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:50.881945 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:50.881962 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:50.882644 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:50.883179 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:50.883188 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:50.883194 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:50.883454 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:50.884578 (+ 1124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:50.884589 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:50.884888 (+ 299us) client_negotiation.cc:769] Sending connection context
0811 03:10:50.885203 (+ 315us) client_negotiation.cc:240] Negotiation successful
0811 03:10:50.885904 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":103,"thread_start_us":446,"threads_started":1}
WARNING: no policy specified for kudu/127.27.109.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.61@KRBTEST.COM" created.
I20260811 03:10:50.890548 21106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:10:50.891813 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:50.891348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34646)
0811 03:10:50.891558 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:50.891686 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260811 03:10:50.892113 21052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
I20260811 03:10:50.893807 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:50.893421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34656)
0811 03:10:50.893579 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:50.893658 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 03:10:50.894035 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:39563: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
Entry for principal kudu/127.27.109.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab.
I20260811 03:10:50.934914 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:39563
--webserver_interface=127.27.109.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38103,127.27.109.61:39563,127.27.109.60:37303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:10:50.943436 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } attempt: 1
I20260811 03:10:50.944809 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:50.944309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34672)
0811 03:10:50.944539 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:50.944673 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260811 03:10:50.945956 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:50.945530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34680)
0811 03:10:50.945698 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:50.945776 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 03:10:50.946184 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:39563: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
I20260811 03:10:51.014516 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } attempt: 2
I20260811 03:10:51.016134 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.015533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34686)
0811 03:10:51.015819 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.015925 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260811 03:10:51.017381 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.016851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34688)
0811 03:10:51.017017 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.017212 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260811 03:10:51.017649 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:39563: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
I20260811 03:10:51.108964 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } attempt: 3
I20260811 03:10:51.110204 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.109744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34700)
0811 03:10:51.109950 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.110054 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 03:10:51.111523 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.111133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34714)
0811 03:10:51.111285 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.111366 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 03:10:51.111923 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:39563: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
I20260811 03:10:51.255265 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } attempt: 4
I20260811 03:10:51.256665 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.256080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34722)
0811 03:10:51.256397 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.256490 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260811 03:10:51.257715 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.257350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34736)
0811 03:10:51.257498 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.257593 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 03:10:51.257916 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.61:39563: Network error: Client connection negotiation failed: client connection to 127.27.109.61:39563: connect: Connection refused (error 111)
W20260811 03:10:51.303479 21113 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 03:10:51.303905 21113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:51.304037 21113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:51.304175 21113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:51.311990 21113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:51.312126 21113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:51.312253 21113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:51.312340 21113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:51.312453 21113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:51.323145 21113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38103,127.27.109.61:39563,127.27.109.60:37303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.61.keytab
--principal=kudu/127.27.109.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.61:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.109.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:51.325102 21113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:51.326601 21113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:51.339486 21118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:51.339460 21119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:51.339421 21121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:51.340477 21113 server_base.cc:1034] running on GCE node
I20260811 03:10:51.341329 21113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:51.342934 21113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:51.344213 21113 hybrid_clock.cc:648] HybridClock initialized: now 1786417851344121 us; error 64 us; skew 500 ppm
I20260811 03:10:51.344587 21113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:51 dist-test-slave-8tdl krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:51.348024 21113 init.cc:377] Logged in from keytab as kudu/127.27.109.61@KRBTEST.COM (short username kudu)
I20260811 03:10:51.349756 21113 webserver.cc:460] Webserver started at http://127.27.109.61:42733/ using document root <none> and password file <none>
I20260811 03:10:51.350500 21113 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:51.350651 21113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:51.351016 21113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:51.354231 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/data/instance:
uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd"
format_stamp: "Formatted at 2026-08-11 03:10:51 on dist-test-slave-8tdl"
server_key: "599f34e47dcc59732d52f1b5116cd79f"
server_key_iv: "45926f2340c24076a776338431281f29"
server_key_version: "encryptionkey@0"
I20260811 03:10:51.355159 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/wal/instance:
uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd"
format_stamp: "Formatted at 2026-08-11 03:10:51 on dist-test-slave-8tdl"
server_key: "599f34e47dcc59732d52f1b5116cd79f"
server_key_iv: "45926f2340c24076a776338431281f29"
server_key_version: "encryptionkey@0"
I20260811 03:10:51.360715 21113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 03:10:51.364364 21129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:51.365975 21113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260811 03:10:51.366230 21113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/wal
uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd"
format_stamp: "Formatted at 2026-08-11 03:10:51 on dist-test-slave-8tdl"
server_key: "599f34e47dcc59732d52f1b5116cd79f"
server_key_iv: "45926f2340c24076a776338431281f29"
server_key_version: "encryptionkey@0"
I20260811 03:10:51.366475 21113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:51.383180 21113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:51.387807 21113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:51.388123 21113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:51.408218 21113 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.61:39563
I20260811 03:10:51.408226 21182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.61:39563 every 8 connection(s)
I20260811 03:10:51.409686 21113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/data/info.pb
I20260811 03:10:51.413296 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 21113
I20260811 03:10:51.413501 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-1/wal/instance
I20260811 03:10:51.413941 21183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:51.414005 28084 external_mini_cluster.cc:1428] Setting key 73b51ece57e673590778db9f3b46fdb5
I20260811 03:10:51.420809 21183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:51 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:10:51.446012 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.423645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.27.109.62:38103)
0811 03:10:51.424252 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:51.424261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:51.425274 (+ 1013us) server_negotiation.cc:407] Connection header received
0811 03:10:51.427409 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:51.427418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:51.427525 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:51.427662 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:51.429352 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.430531 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.433201 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.433642 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.433743 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:51.438814 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:51.438820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:51.438824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:51.438883 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:51.442151 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:51.443310 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:51.443316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:51.443318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:51.443460 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:51.443976 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:51.443979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:51.443982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:51.444317 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:51.444506 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:51.445309 (+ 803us) server_negotiation.cc:299] Negotiation successful
0811 03:10:51.445576 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":291,"threads_started":1}
I20260811 03:10:51.447135 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.424238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38103 (local address 127.0.0.1:49548)
0811 03:10:51.425077 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.425125 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:51.427109 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:51.427884 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:51.427892 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:51.428477 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:51.429091 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:51.429111 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.430756 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.430765 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:51.432993 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:51.433015 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.433909 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.433919 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:51.434271 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:51.435212 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:51.435248 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:51.438408 (+ 3160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:51.442402 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:51.442415 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:51.442447 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:51.443136 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:51.443587 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:51.443593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:51.443597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:51.443820 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:51.444474 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:51.444484 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:51.444906 (+ 422us) client_negotiation.cc:769] Sending connection context
0811 03:10:51.445198 (+ 292us) client_negotiation.cc:240] Negotiation successful
0811 03:10:51.445863 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":354,"threads_started":1}
I20260811 03:10:51.449316 21183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.109.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.109.60@KRBTEST.COM" created.
Aug 11 03:10:51 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:10:51.464776 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.450159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34744)
0811 03:10:51.450341 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.450364 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:51.450475 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:51.451647 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:51.451652 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:51.452018 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:51.452226 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:51.452242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.453754 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.453762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:51.454600 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:51.454612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.455490 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.455496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:51.455686 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:51.456313 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:51.456331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:51.458789 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:51.462130 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:51.462135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:51.462139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:51.462499 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:51.462937 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:51.462943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:51.462946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:51.463091 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:51.463717 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:51.463725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:51.463878 (+ 153us) client_negotiation.cc:769] Sending connection context
0811 03:10:51.464019 (+ 141us) client_negotiation.cc:240] Negotiation successful
0811 03:10:51.464317 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260811 03:10:51.464819 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.450452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.27.109.61:39563)
0811 03:10:51.451173 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:51.451179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:51.451208 (+ 29us) server_negotiation.cc:407] Connection header received
0811 03:10:51.451255 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:51.451261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:51.451450 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:51.451576 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:51.452417 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.453621 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.454769 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.455335 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.455469 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:51.458997 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:51.459011 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:51.459026 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:51.459070 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:51.461971 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:51.462650 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:51.462658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:51.462660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:51.462828 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:51.463193 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:51.463200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:51.463203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:51.463570 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:51.463798 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:51.464198 (+ 400us) server_negotiation.cc:299] Negotiation successful
0811 03:10:51.464410 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":345,"threads_started":1}
I20260811 03:10:51.467391 21183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:10:51.468905 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.468399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57188)
0811 03:10:51.468606 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.468732 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260811 03:10:51.469260 21132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:51.471055 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.470562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57200)
0811 03:10:51.470773 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.470890 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 03:10:51.471364 21183 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
Entry for principal kudu/127.27.109.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab.
I20260811 03:10:51.497996 28084 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
/tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:37303
--webserver_interface=127.27.109.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.109.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.109.62:38103,127.27.109.61:39563,127.27.109.60:37303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 03:10:51.520749 21183 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 1
I20260811 03:10:51.522233 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.521644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57210)
0811 03:10:51.521888 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.522056 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260811 03:10:51.523516 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.523133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57222)
0811 03:10:51.523287 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.523396 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260811 03:10:51.523834 21183 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:51.557171 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } attempt: 5
Aug 11 03:10:51 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417850, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:10:51.571550 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.557974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34752)
0811 03:10:51.558172 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.558194 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:51.558352 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:51.558777 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:51.558783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:51.559212 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:51.559398 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:51.559408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.560456 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.560462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:51.561340 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:51.561352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.562147 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.562153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:51.562386 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:51.563106 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:51.563126 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:51.565564 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:51.568866 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:51.568872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:51.568877 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:51.569278 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:51.569682 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:51.569687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:51.569691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:51.569846 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:51.570438 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:51.570443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:51.570619 (+ 176us) client_negotiation.cc:769] Sending connection context
0811 03:10:51.570861 (+ 242us) client_negotiation.cc:240] Negotiation successful
0811 03:10:51.571180 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 03:10:51.571553 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.558199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.27.109.61:39563)
0811 03:10:51.558422 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:51.558428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:51.558450 (+ 22us) server_negotiation.cc:407] Connection header received
0811 03:10:51.558519 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:51.558523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:51.558606 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:51.558735 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:51.559567 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.560308 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.561513 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:51.561979 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:51.562131 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:51.565799 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:51.565804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:51.565808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:51.565845 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:51.568690 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:51.569428 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:51.569432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:51.569434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:51.569564 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:51.569970 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:51.569977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:51.569984 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:51.570295 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:51.570480 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:51.570961 (+ 481us) server_negotiation.cc:299] Negotiation successful
0811 03:10:51.571202 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 03:10:51.572611 21106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:10:51.573683 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.573263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57228)
0811 03:10:51.573434 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.573531 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 03:10:51.574607 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.574277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57230)
0811 03:10:51.574420 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.574505 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 03:10:51.574854 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:51.592103 21183 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 2
I20260811 03:10:51.593401 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.592921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57244)
0811 03:10:51.593139 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.593236 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 03:10:51.594512 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.594169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57250)
0811 03:10:51.594319 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.594390 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 03:10:51.594760 21183 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:51.626106 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 1
I20260811 03:10:51.627470 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.626990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57266)
0811 03:10:51.627208 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.627303 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260811 03:10:51.628501 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.628150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57272)
0811 03:10:51.628305 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.628380 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 03:10:51.628808 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:51.685997 21183 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 3
I20260811 03:10:51.687294 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.686786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57286)
0811 03:10:51.687027 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.687128 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 03:10:51.688499 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.688038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57292)
0811 03:10:51.688308 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.688385 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260811 03:10:51.688762 21183 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:51.697094 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 2
I20260811 03:10:51.698433 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.697864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57306)
0811 03:10:51.698133 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.698230 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260811 03:10:51.699568 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.699197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57314)
0811 03:10:51.699361 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.699444 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 03:10:51.699797 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:51.806025 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 3
I20260811 03:10:51.807231 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.806752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57318)
0811 03:10:51.806953 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.807051 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 03:10:51.808327 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.807877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57334)
0811 03:10:51.808143 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.808225 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260811 03:10:51.808521 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:51.831959 21183 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 4
I20260811 03:10:51.833295 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.832737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57348)
0811 03:10:51.832984 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.833112 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260811 03:10:51.834348 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.834024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57358)
0811 03:10:51.834176 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.834251 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 03:10:51.834550 21183 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
W20260811 03:10:51.881488 21192 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 03:10:51.881939 21192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 03:10:51.882081 21192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 03:10:51.882223 21192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 03:10:51.890192 21192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 03:10:51.890358 21192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 03:10:51.890491 21192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 03:10:51.890583 21192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 03:10:51.890723 21192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 03:10:51.901453 21192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.109.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.109.62:38103,127.27.109.61:39563,127.27.109.60:37303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/krb5kdc/kudu_127.27.109.60.keytab
--principal=kudu/127.27.109.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.109.60:37303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.109.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 02:07:30 UTC on ccycloud.quasar-qylciq.root.comops.site
build id 4579572
ASAN enabled
I20260811 03:10:51.903467 21192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 03:10:51.905179 21192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 03:10:51.918113 21199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:51.918170 21202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 03:10:51.918123 21200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 03:10:51.919090 21192 server_base.cc:1034] running on GCE node
I20260811 03:10:51.920002 21192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 03:10:51.921587 21192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 03:10:51.922814 21192 hybrid_clock.cc:648] HybridClock initialized: now 1786417851922716 us; error 63 us; skew 500 ppm
I20260811 03:10:51.923187 21192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 03:10:51 dist-test-slave-8tdl krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 03:10:51.926870 21192 init.cc:377] Logged in from keytab as kudu/127.27.109.60@KRBTEST.COM (short username kudu)
I20260811 03:10:51.928453 21192 webserver.cc:460] Webserver started at http://127.27.109.60:46809/ using document root <none> and password file <none>
I20260811 03:10:51.929231 21192 fs_manager.cc:359] Metadata directory not provided
I20260811 03:10:51.929378 21192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 03:10:51.929728 21192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 03:10:51.932977 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/data/instance:
uuid: "de71455e1a424c45b4374656afd8a548"
format_stamp: "Formatted at 2026-08-11 03:10:51 on dist-test-slave-8tdl"
server_key: "1c71a0ec9f3c3bb751da56b1606e3673"
server_key_iv: "711c97ed088ab875cd763c2c78b200b7"
server_key_version: "encryptionkey@0"
I20260811 03:10:51.933838 21192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/wal/instance:
uuid: "de71455e1a424c45b4374656afd8a548"
format_stamp: "Formatted at 2026-08-11 03:10:51 on dist-test-slave-8tdl"
server_key: "1c71a0ec9f3c3bb751da56b1606e3673"
server_key_iv: "711c97ed088ab875cd763c2c78b200b7"
server_key_version: "encryptionkey@0"
I20260811 03:10:51.939358 21192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 03:10:51.943266 21209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 03:10:51.944854 21192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 03:10:51.945086 21192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/data,/tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/wal
uuid: "de71455e1a424c45b4374656afd8a548"
format_stamp: "Formatted at 2026-08-11 03:10:51 on dist-test-slave-8tdl"
server_key: "1c71a0ec9f3c3bb751da56b1606e3673"
server_key_iv: "711c97ed088ab875cd763c2c78b200b7"
server_key_version: "encryptionkey@0"
I20260811 03:10:51.945340 21192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 03:10:51.979077 21192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 03:10:51.983945 21192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 03:10:51.984258 21192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 03:10:51.985796 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 4
I20260811 03:10:51.987226 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.986590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57364)
0811 03:10:51.986899 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.987079 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260811 03:10:51.988323 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:51.987955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57378)
0811 03:10:51.988112 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:51.988192 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 03:10:51.988545 21106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.109.60:37303: Network error: Client connection negotiation failed: client connection to 127.27.109.60:37303: connect: Connection refused (error 111)
I20260811 03:10:52.007037 21192 rpc_server.cc:307] RPC server started. Bound to: 127.27.109.60:37303
I20260811 03:10:52.007098 21268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.109.60:37303 every 8 connection(s)
I20260811 03:10:52.008942 21192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/data/info.pb
I20260811 03:10:52.013751 21269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 03:10:52.017220 28084 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu as pid 21192
I20260811 03:10:52.017598 28084 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWXDOoZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786417454123933-28084-0/minicluster-data/master-2/wal/instance
I20260811 03:10:52.018164 28084 external_mini_cluster.cc:1428] Setting key 365b8ac6b516119d7bf07c9b4a441c59
I20260811 03:10:52.022367 21269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:52 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 11 03:10:52 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:10:52.045051 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.026773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.27.109.62:38103)
0811 03:10:52.027381 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.027389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.027900 (+ 511us) server_negotiation.cc:407] Connection header received
0811 03:10:52.028133 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.028137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.028218 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.028349 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.029911 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.030977 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.033006 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.033427 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.033598 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.037665 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.037675 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.037680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.037743 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.041175 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.042055 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.042062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.042066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.042219 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.042747 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.042752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.042755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.043080 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.043233 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.044314 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.044569 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":335,"threads_started":1}
I20260811 03:10:52.045570 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.020491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.27.109.60:37303)
0811 03:10:52.021385 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.021398 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.021447 (+ 49us) server_negotiation.cc:407] Connection header received
0811 03:10:52.023342 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.023368 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.023837 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.024268 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.026236 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.028434 (+ 2198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.030247 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.031294 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.031448 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.036562 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.036576 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.036598 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.036632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.040784 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.041752 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.041758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.041760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.041995 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.042431 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.042434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.042436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.043212 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.043452 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.043893 (+ 441us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.044302 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"spinlock_wait_cycles":18816,"thread_start_us":418,"threads_started":1}
I20260811 03:10:52.045720 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.026801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.62:38103 (local address 127.0.0.1:49554)
0811 03:10:52.027742 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:52.027787 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:52.027985 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:52.028415 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:52.028424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:52.029370 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:52.029729 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.029745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.031165 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.031169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.032815 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.032834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.033713 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.033722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.033953 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.034637 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:52.034693 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:52.037416 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:52.041321 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.041333 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.041347 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.041929 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.042341 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.042348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.042354 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.042599 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.043218 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:52.043224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:52.043496 (+ 272us) client_negotiation.cc:769] Sending connection context
0811 03:10:52.043846 (+ 350us) client_negotiation.cc:240] Negotiation successful
0811 03:10:52.044561 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":251,"spinlock_wait_cycles":896,"thread_start_us":551,"threads_started":1}
I20260811 03:10:52.047515 21269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 03:10:52 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
Aug 11 03:10:52 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.62@KRBTEST.COM
I20260811 03:10:52.062769 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.048261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.61:39563 (local address 127.0.0.1:34756)
0811 03:10:52.048447 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:52.048469 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:52.048615 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:52.049018 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:52.049022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:52.049576 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:52.049728 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.049760 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.050933 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.050939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.051975 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.051994 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.052706 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.052714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.052932 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.053587 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:52.053632 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:52.056354 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:52.060021 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.060032 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.060035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.060423 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.060843 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.060848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.060856 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.061003 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.061708 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:52.061713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:52.061882 (+ 169us) client_negotiation.cc:769] Sending connection context
0811 03:10:52.062034 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 03:10:52.062356 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 03:10:52.062824 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.048416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.27.109.61:39563)
0811 03:10:52.048604 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.048610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.048632 (+ 22us) server_negotiation.cc:407] Connection header received
0811 03:10:52.048763 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.048767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.048831 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.048995 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.049913 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.050773 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.052134 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.052547 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.052649 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.056577 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.056584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.056589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.056631 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.059862 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.060559 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.060568 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.060571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.060722 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.061114 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.061119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.061125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.061398 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.061756 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.062206 (+ 450us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.062433 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 03:10:52.063920 21269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } has no permanent_uuid. Determining permanent_uuid...
I20260811 03:10:52.064697 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.049368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.27.109.62:38103)
0811 03:10:52.049527 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.049531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.049544 (+ 13us) server_negotiation.cc:407] Connection header received
0811 03:10:52.049592 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.049595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.049658 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.049777 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.051246 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.051899 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.053493 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.054054 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.054155 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.058842 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.058847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.058851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.058886 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.061673 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.062396 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.062401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.062403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.062528 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.063010 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.063017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.063021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.063392 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.063696 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.064109 (+ 413us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.064335 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Aug 11 03:10:52 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.60@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
Aug 11 03:10:52 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.109.61@KRBTEST.COM
I20260811 03:10:52.080554 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.064763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57390)
0811 03:10:52.065126 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:52.065141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:52.065220 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:52.065531 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:52.065536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:52.065983 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:52.066138 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.066146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.067458 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.067463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.068934 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.068948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.069890 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.069894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.070172 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.070818 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:52.070845 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:52.073764 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:52.077180 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.077188 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.077195 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.077774 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.078348 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.078352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.078355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.078568 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.079483 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:52.079486 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:52.079653 (+ 167us) client_negotiation.cc:769] Sending connection context
0811 03:10:52.079773 (+ 120us) client_negotiation.cc:240] Negotiation successful
0811 03:10:52.080103 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":39}
I20260811 03:10:52.080816 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.064997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.27.109.60:37303)
0811 03:10:52.065166 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.065172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.065191 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:10:52.065307 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.065315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.065382 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.065474 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.066278 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.067323 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.069080 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.069739 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.069875 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.073968 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.073975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.073980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.074022 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.077010 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.078017 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.078023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.078026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.078194 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.078741 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.078749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.078756 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.079327 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.079501 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.079931 (+ 430us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.080375 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":57}
I20260811 03:10:52.085085 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.066834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.27.109.61:39563)
0811 03:10:52.067002 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.067008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.067026 (+ 18us) server_negotiation.cc:407] Connection header received
0811 03:10:52.067184 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.067189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.067260 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.067370 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.069092 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.070264 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.072296 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.072987 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.073140 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.078082 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.078089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.078097 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.078148 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.082016 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.082950 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.082955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.082957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.083103 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.083553 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.083558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.083560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.083848 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.084130 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.084510 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.084763 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 03:10:52.086578 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: Bootstrap starting.
I20260811 03:10:52.090550 21269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:52.090605 28084 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 03:10:52.091741 21269 log.cc:824] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:52.094957 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: No bootstrap required, opened a new log
I20260811 03:10:52.093221 21080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dcce9e26f9234ebaa793367eb2aab8f8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49564
I20260811 03:10:52.102576 21269 raft_consensus.cc:348] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } }
I20260811 03:10:52.103124 21269 raft_consensus.cc:729] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de71455e1a424c45b4374656afd8a548, State: Initialized, Role: FOLLOWER
I20260811 03:10:52.103641 21269 consensus_queue.cc:260] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } }
I20260811 03:10:52.104871 21276 sys_catalog.cc:455] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.105360 21276 sys_catalog.cc:458] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:52.106351 21269 sys_catalog.cc:564] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 03:10:52.117496 21289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:10:52.117672 21289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:10:52.114765 21192 master_runner.cc:186] Error getting master registration for 127.27.109.62:38103: OK, instance_id { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" instance_seqno: 1786417850835821 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 03:10:52.123721 21192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 03:10:52.133775 21183 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 5
Aug 11 03:10:52 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.61@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260811 03:10:52.152242 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.135020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.27.109.60:37303)
0811 03:10:52.135240 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.135245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.135264 (+ 19us) server_negotiation.cc:407] Connection header received
0811 03:10:52.135325 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.135329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.135393 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.135484 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.136589 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.137880 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.139651 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.140344 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.140710 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.144891 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.144898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.144902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.144943 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.148771 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.149839 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.149845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.149848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.150022 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.150499 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.150504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.150507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.150984 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.151179 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.151584 (+ 405us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.151842 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 03:10:52.152235 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.134585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57396)
0811 03:10:52.134868 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:52.134897 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:52.135042 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:52.135574 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:52.135579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:52.136138 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:52.136372 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.136398 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.138052 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.138059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.139322 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.139344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.140729 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.140736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.141004 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.141723 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:52.141743 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:52.144611 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:52.149030 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.149035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.149038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.149574 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.150178 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.150183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.150187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.150361 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.151127 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:52.151132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:52.151309 (+ 177us) client_negotiation.cc:769] Sending connection context
0811 03:10:52.151454 (+ 145us) client_negotiation.cc:240] Negotiation successful
0811 03:10:52.151824 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260811 03:10:52.159837 21183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: Bootstrap starting.
I20260811 03:10:52.164297 21183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:52.165710 21183 log.cc:824] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:52.169528 21183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: No bootstrap required, opened a new log
I20260811 03:10:52.177489 21183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } }
I20260811 03:10:52.177879 21183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6ba9cd2d634a338c8bf5621cb0fabd, State: Initialized, Role: FOLLOWER
I20260811 03:10:52.178424 21183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } }
I20260811 03:10:52.179548 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.180011 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:52.180831 21183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [sys.catalog]: configured and running, proceeding with master startup.
W20260811 03:10:52.192461 21304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 03:10:52.192713 21304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 03:10:52.189129 21113 master_runner.cc:186] Error getting master registration for 127.27.109.62:38103: OK, instance_id { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" instance_seqno: 1786417850835821 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 03:10:52.201236 21113 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 03:10:52.236836 21215 tablet.cc:2369] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:10:52.265812 21106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } attempt: 5
Aug 11 03:10:52 dist-test-slave-8tdl krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786417850, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.109.62@KRBTEST.COM for kudu/127.27.109.60@KRBTEST.COM
I20260811 03:10:52.286557 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.266768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.109.60:37303 (local address 127.0.0.1:57412)
0811 03:10:52.267114 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0811 03:10:52.267139 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 03:10:52.267285 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 03:10:52.268012 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 03:10:52.268019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 03:10:52.268613 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0811 03:10:52.268841 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.268856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.270606 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.270612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.272210 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 03:10:52.272239 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.273440 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.273446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 03:10:52.273673 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.274449 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 03:10:52.274474 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 03:10:52.277747 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 03:10:52.283123 (+ 5376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.283129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.283134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.283786 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.284336 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 03:10:52.284342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 03:10:52.284346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 03:10:52.284624 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 03:10:52.285411 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 03:10:52.285416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 03:10:52.285617 (+ 201us) client_negotiation.cc:769] Sending connection context
0811 03:10:52.285777 (+ 160us) client_negotiation.cc:240] Negotiation successful
0811 03:10:52.286107 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260811 03:10:52.286700 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.266855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.27.109.60:37303)
0811 03:10:52.267572 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.267579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.267602 (+ 23us) server_negotiation.cc:407] Connection header received
0811 03:10:52.267679 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.267686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.267825 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.267941 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.269049 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.270433 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.272387 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.273234 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.273394 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.278088 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.278097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.278104 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.278162 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.282814 (+ 4652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.283997 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.284004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.284009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.284212 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.284813 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.284820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.284824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.285240 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.285453 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.285937 (+ 484us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.286236 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540}
I20260811 03:10:52.294023 21106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8: Bootstrap starting.
I20260811 03:10:52.298931 21106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8: Neither blocks nor log segments found. Creating new log.
I20260811 03:10:52.300592 21106 log.cc:824] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8: Log is configured to *not* fsync() on all Append() calls
I20260811 03:10:52.305121 21106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8: No bootstrap required, opened a new log
I20260811 03:10:52.313758 21106 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } }
I20260811 03:10:52.314244 21106 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcce9e26f9234ebaa793367eb2aab8f8, State: Initialized, Role: FOLLOWER
I20260811 03:10:52.314908 21106 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } }
I20260811 03:10:52.316401 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.317167 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:52.317628 21106 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 03:10:52.317754 21080 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 03:10:52.317960 21080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:52.321537 21080 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } }
W20260811 03:10:52.323289 21058 tablet.cc:2369] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:10:52.324357 21157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd"
I20260811 03:10:52.324893 21157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:52.328896 21157 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcce9e26f9234ebaa793367eb2aab8f8 in term 1.
I20260811 03:10:52.329916 21052 leader_election.cc:302] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6ba9cd2d634a338c8bf5621cb0fabd, dcce9e26f9234ebaa793367eb2aab8f8; no voters:
I20260811 03:10:52.333336 21239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "de71455e1a424c45b4374656afd8a548"
I20260811 03:10:52.333952 21239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [term 0 FOLLOWER]: Advancing to term 1
I20260811 03:10:52.337765 21239 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcce9e26f9234ebaa793367eb2aab8f8 in term 1.
I20260811 03:10:52.338740 21080 leader_election.cc:288] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [CANDIDATE]: Term 1 election: Requested vote from peers 4e6ba9cd2d634a338c8bf5621cb0fabd (127.27.109.61:39563), de71455e1a424c45b4374656afd8a548 (127.27.109.60:37303)
I20260811 03:10:52.345914 21307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 03:10:52.359413 21307 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [term 1 LEADER]: Becoming Leader. State: Replica: dcce9e26f9234ebaa793367eb2aab8f8, State: Running, Role: LEADER
I20260811 03:10:52.360304 21307 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } }
I20260811 03:10:52.378109 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcce9e26f9234ebaa793367eb2aab8f8. Latest consensus state: current_term: 1 leader_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.378885 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:52.390376 21325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260811 03:10:52.391016 21135 tablet.cc:2369] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 03:10:52.394277 21328 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 03:10:52.418493 21325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 03:10:52.419332 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.372555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.27.109.62:38103)
0811 03:10:52.372819 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.372829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.383485 (+ 10656us) server_negotiation.cc:407] Connection header received
0811 03:10:52.383872 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.383881 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.383998 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.384091 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.395024 (+ 10933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.396142 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.399182 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.399922 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.400027 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.403725 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.403736 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.403741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.403806 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.414928 (+ 11122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.416017 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.416025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.416031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.416215 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.416747 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.416753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.416758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.417217 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.418454 (+ 1237us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.418598 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.418866 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 03:10:52.426893 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.375101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.27.109.60:37303)
0811 03:10:52.375325 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.375332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.387637 (+ 12305us) server_negotiation.cc:407] Connection header received
0811 03:10:52.387974 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.387980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.388082 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.388177 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.402849 (+ 14672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.403974 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.406241 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.407031 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.407193 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.418366 (+ 11173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.418374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.418398 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.418444 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.422575 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.423657 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.423664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.423667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.423826 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.424299 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.424305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.424309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.424741 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.425029 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.426172 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.426443 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 03:10:52.441643 21239 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [term 1 FOLLOWER]: Refusing update from remote peer dcce9e26f9234ebaa793367eb2aab8f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:10:52.442471 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:52.387716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.27.109.61:39563)
0811 03:10:52.387966 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.387976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.395726 (+ 7750us) server_negotiation.cc:407] Connection header received
0811 03:10:52.395915 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.395921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.396020 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.396125 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.404885 (+ 8760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.406122 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.427438 (+ 21316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:52.428245 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:52.428378 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:52.432112 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 03:10:52.432119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 03:10:52.432125 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 03:10:52.432175 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 03:10:52.436289 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.439458 (+ 3169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.439469 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.439474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.439656 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 03:10:52.440230 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 03:10:52.440235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 03:10:52.440238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 03:10:52.440688 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 03:10:52.440974 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:52.441619 (+ 645us) server_negotiation.cc:299] Negotiation successful
0811 03:10:52.442098 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 03:10:52.442920 21314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:10:52.444830 21157 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [term 1 FOLLOWER]: Refusing update from remote peer dcce9e26f9234ebaa793367eb2aab8f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 03:10:52.445902 21314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 03:10:52.467314 21276 sys_catalog.cc:455] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcce9e26f9234ebaa793367eb2aab8f8. Latest consensus state: current_term: 1 leader_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.467700 21276 sys_catalog.cc:458] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:52.470338 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcce9e26f9234ebaa793367eb2aab8f8. Latest consensus state: current_term: 1 leader_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.470669 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:52.482043 21276 sys_catalog.cc:455] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.482393 21276 sys_catalog.cc:458] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:52.484360 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.484653 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 03:10:52.485173 21332 mvcc.cc:205] Tried to move back new op lower bound from 7317167523559251968 to 7317167523286749184. Current Snapshot: MvccSnapshot[applied={T|T < 7317167523559251968}]
I20260811 03:10:52.487325 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.487658 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:52.488029 21325 catalog_manager.cc:1347] Generated new cluster ID: a02e8194b35c4f79a627bff8da7512ac
I20260811 03:10:52.488119 21325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 03:10:52.491720 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcce9e26f9234ebaa793367eb2aab8f8" member_type: VOTER last_known_addr { host: "127.27.109.62" port: 38103 } } peers { permanent_uuid: "4e6ba9cd2d634a338c8bf5621cb0fabd" member_type: VOTER last_known_addr { host: "127.27.109.61" port: 39563 } } peers { permanent_uuid: "de71455e1a424c45b4374656afd8a548" member_type: VOTER last_known_addr { host: "127.27.109.60" port: 37303 } } }
I20260811 03:10:52.492127 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8 [sys.catalog]: This master's current role is: LEADER
I20260811 03:10:52.527323 21325 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 03:10:52.529353 21325 catalog_manager.cc:1504] Loading token signing keys...
I20260811 03:10:52.570859 21325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcce9e26f9234ebaa793367eb2aab8f8: Generated new TSK 0
I20260811 03:10:52.572000 21325 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 03:10:52.742051 21273 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.716568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.27.109.62:38103)
0811 03:10:52.716834 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.716841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.737499 (+ 20658us) server_negotiation.cc:407] Connection header received
0811 03:10:52.740399 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.740406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.740508 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.740583 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.741778 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49588: BlockingRecv error: recv got EOF from 127.0.0.1:49588 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260811 03:10:52.744585 21272 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.722901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.27.109.60:37303)
0811 03:10:52.723140 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.723148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.739353 (+ 16205us) server_negotiation.cc:407] Connection header received
0811 03:10:52.742641 (+ 3288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.742650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.742801 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.742903 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.744285 (+ 1382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57420: BlockingRecv error: recv got EOF from 127.0.0.1:57420 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260811 03:10:52.747762 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.738480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.27.109.61:39563)
0811 03:10:52.738742 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.738748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.745418 (+ 6670us) server_negotiation.cc:407] Connection header received
0811 03:10:52.746097 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.746103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.746221 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.746295 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.747489 (+ 1194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34786: BlockingRecv error: recv got EOF from 127.0.0.1:34786 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260811 03:10:52.769385 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.766855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.27.109.61:39563)
0811 03:10:52.767134 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.767143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.767687 (+ 544us) server_negotiation.cc:407] Connection header received
0811 03:10:52.767982 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.767987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.768099 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.768235 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.769053 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34794: BlockingRecv error: recv got EOF from 127.0.0.1:34794 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260811 03:10:52.777349 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.774549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.27.109.61:39563)
0811 03:10:52.774870 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.774880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.775488 (+ 608us) server_negotiation.cc:407] Connection header received
0811 03:10:52.775847 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.775854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.776018 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.776157 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.777088 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34798: BlockingRecv error: recv got EOF from 127.0.0.1:34798 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260811 03:10:52.792855 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.790285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.27.109.61:39563)
0811 03:10:52.790588 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.790597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.791236 (+ 639us) server_negotiation.cc:407] Connection header received
0811 03:10:52.791476 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.791482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.791596 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.791736 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.792579 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34802: BlockingRecv error: recv got EOF from 127.0.0.1:34802 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260811 03:10:52.803710 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.801037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.27.109.61:39563)
0811 03:10:52.801383 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.801393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.801979 (+ 586us) server_negotiation.cc:407] Connection header received
0811 03:10:52.802226 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.802232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.802322 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.802438 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.803461 (+ 1023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34810: BlockingRecv error: recv got EOF from 127.0.0.1:34810 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
W20260811 03:10:52.818444 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.815982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.27.109.61:39563)
0811 03:10:52.816281 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.816291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.816885 (+ 594us) server_negotiation.cc:407] Connection header received
0811 03:10:52.817143 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.817148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.817229 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.817344 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.818175 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34822: BlockingRecv error: recv got EOF from 127.0.0.1:34822 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260811 03:10:52.847853 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.845096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.27.109.61:39563)
0811 03:10:52.845288 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.845295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.845878 (+ 583us) server_negotiation.cc:407] Connection header received
0811 03:10:52.846184 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.846190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.846289 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.846361 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.847614 (+ 1253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34826: BlockingRecv error: recv got EOF from 127.0.0.1:34826 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260811 03:10:52.887130 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.884551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.27.109.61:39563)
0811 03:10:52.884840 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.884848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.885314 (+ 466us) server_negotiation.cc:407] Connection header received
0811 03:10:52.885538 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.885542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.885640 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.885741 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.886814 (+ 1073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34836: BlockingRecv error: recv got EOF from 127.0.0.1:34836 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260811 03:10:52.957599 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:52.955381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.27.109.61:39563)
0811 03:10:52.955619 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:52.955628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:52.956036 (+ 408us) server_negotiation.cc:407] Connection header received
0811 03:10:52.956296 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:52.956302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:52.956397 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:52.956517 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:52.957348 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34848: BlockingRecv error: recv got EOF from 127.0.0.1:34848 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260811 03:10:53.092608 21188 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 03:10:53.090438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.27.109.61:39563)
0811 03:10:53.090733 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:53.090742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:53.091098 (+ 356us) server_negotiation.cc:407] Connection header received
0811 03:10:53.091348 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:53.091354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:53.091474 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:53.091595 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 03:10:53.092406 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34864: BlockingRecv error: recv got EOF from 127.0.0.1:34864 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260811 03:10:53.120137 21289 catalog_manager.cc:1259] Loaded cluster ID: a02e8194b35c4f79a627bff8da7512ac
I20260811 03:10:53.120277 21289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: loading cluster ID for follower catalog manager: success
I20260811 03:10:53.122987 21289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: acquiring CA information for follower catalog manager: success
I20260811 03:10:53.124905 21289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de71455e1a424c45b4374656afd8a548: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 03:10:53.195081 21304 catalog_manager.cc:1259] Loaded cluster ID: a02e8194b35c4f79a627bff8da7512ac
I20260811 03:10:53.195199 21304 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: loading cluster ID for follower catalog manager: success
I20260811 03:10:53.197959 21304 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: acquiring CA information for follower catalog manager: success
I20260811 03:10:53.200047 21304 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e6ba9cd2d634a338c8bf5621cb0fabd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 03:10:53.360654 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:53.354301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.27.109.61:39563)
0811 03:10:53.354526 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:53.354535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:53.354871 (+ 336us) server_negotiation.cc:407] Connection header received
0811 03:10:53.355176 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:53.355182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:53.355358 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:53.355490 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:53.355955 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.356693 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.358197 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.358624 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.358810 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:53.359361 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:53.359971 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:53.360150 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:53.360253 (+ 103us) server_negotiation.cc:299] Negotiation successful
0811 03:10:53.360349 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 03:10:53.374315 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:53.367519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.27.109.60:37303)
0811 03:10:53.368064 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:53.368069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:53.368183 (+ 114us) server_negotiation.cc:407] Connection header received
0811 03:10:53.368362 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:53.368367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:53.368564 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:53.368730 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:53.369214 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.370090 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.371744 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.372229 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.372351 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:53.372970 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:53.373606 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:53.373728 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:53.373835 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 03:10:53.373915 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":307,"threads_started":1}
I20260811 03:10:53.389283 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:53.381477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.27.109.60:37303)
0811 03:10:53.381762 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:53.381771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:53.382665 (+ 894us) server_negotiation.cc:407] Connection header received
0811 03:10:53.383049 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:53.383055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:53.383154 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:53.383270 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:53.383991 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.385158 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.387234 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.387683 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.387846 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:53.388323 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:53.388445 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:53.388544 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:53.388708 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 03:10:53.388799 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260811 03:10:53.391042 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:53.382534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.27.109.61:39563)
0811 03:10:53.382761 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:53.382767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:53.383359 (+ 592us) server_negotiation.cc:407] Connection header received
0811 03:10:53.383660 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:53.383667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:53.383777 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:53.383846 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:53.384752 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.385747 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.388353 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.389066 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.389183 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:53.390109 (+ 926us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:53.390261 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:53.390371 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:53.390559 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 03:10:53.390661 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 03:10:53.391652 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 03:10:53.381228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.27.109.62:38103)
0811 03:10:53.381767 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0811 03:10:53.381773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 03:10:53.382573 (+ 800us) server_negotiation.cc:407] Connection header received
0811 03:10:53.383029 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 03:10:53.383037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 03:10:53.383301 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 03:10:53.383498 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 03:10:53.384510 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.385565 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.388352 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 03:10:53.389056 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 03:10:53.389192 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 03:10:53.390171 (+ 979us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 03:10:53.390710 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 03:10:53.390828 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 03:10:53.391298 (+ 470us) server_negotiation.cc:299] Negotiation successful
0811 03:10:53.391406 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":278,"threads_started":1}
I20260811 03:10:53.397380 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 21033
I20260811 03:10:53.398833 21101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:53.514145 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 21033
I20260811 03:10:53.543589 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 21113
I20260811 03:10:53.544876 21178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:53.658802 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 21113
I20260811 03:10:53.684772 28084 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu pid 21192
I20260811 03:10:53.686022 21263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 03:10:53.803702 28084 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWXDOoZ/build/asan/bin/kudu with pid 21192
2026-08-11T03:10:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5067 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10242 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (399642 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST